Trace: » testing.html
Table of Contents

[hemmerling] Testing

Related pages:

Organizations

Fairs, Conferences, Events

Certification

Testing Experts

Crowdtesting Jobs

Basics

  • Explorative testing = Manual testing, with testplan.

Questions about the Testing of Embedded Software

  1. How to test embedded software ? Under the conditions
    1. Most of the code is machine-dependent, i.e. a function to be tested accesses hardware ( reads and writes registers, while the results depend on external conditions and events ).
    2. Resources ( RAM,.. ) are limited, in comparison to the well-known standard of deskop PCs.
    3. Use of standard ( free, OpenSource, commercial )Test frameworks and especially Unit Test frameworks.

Mockup - Simulation of Objects

  • See [unittest.html|Unit Test / Module Test]].

Unit Test vs Functional Test

Keyword-based Testing

Test-First Approach - UX-Driven Testing

Test-First Approach

  • The domain experts define the input mask of screens, of an IT application, plus the behaviour ( how to move from one screen to the next ), by simple tools, e.g. spreadsheets. The software developers are not involved.
  • If this is derived from the specification, it can be used also as guide for implementation by the software developers.
  • This is used also later for integration test.
  • Alternatively, the domain expert might build a rapid prototype, with little or no help by a software developer.

UX-Driven Testing

  • The software is developed from point of view of the user, and his / her “user experience” ( UX ).

Property-based Testing

Scala ( and Java )

Haskell

F#

Javascript

Instrumentation Frameworks for Dynamic Application Testing

Test Mangement, Functional Test Tools, Automation Frameworks, GUI Testing Tools / Capture & Replay Tools

Important Dynamic Code Analysis Tools / Tools for Dynamic Analysis Security Testing ( DAST ), according to VDC Research in 2015, 2016-08, 2018-08

  • RapiTime/RapiCover (Rapita Systems).
  • LDRA Testbed (LDRA).
  • SilkTest (Borland / Micro Focus).
  • HP.
    • HP Quality Center / ALM.
    • HP Security Fortify (HP).
  • Rational Test RealTime/Tau Tester (IBM Rational).
  • TestQuest (BSQUARE).
  • Cantata++ /AdaTEST (QA Systems).
  • Test Center (Keynote/DeviceAnywhere).
  • Parasoft C++test/Insure++/Standard Compliance (Parasoft).
  • MobileCloud (Perfecto Mobile).
  • Coverity.
    • Coverity Quality Adviser (Coverity) 2015-2016.
    • Coverity Dynamic Analysis (Coverity) 2018.
  • VectorCAST (Vector Software).
  • Veracode (Veracode), 2016-08
  • SW-Assure (Monroe Software), 2016-08.

Some other General Test Management Tools

Testdata Creation and Testdata Management

Methods

  • EN.Wikipedia "Fuzzing", DE.Wikipedia "Fuzzing" - “Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks”.

Software

  • The commercial testdata management system Averna Proligent.
  • The OpenSource Damhave Systems "sequence" - “A UNIX/Linux command line tool for generation binary or ASCII test sequences. Sequence is specially useful in bash scripts for test data generation when testing and debugging your own linux device drivers or playing with flash or MTD devices”.
  • The OpenSource GenerateData.com - Open-source data generator script.

Integration & Automation Frameworks, Tools for functional Tests and Regression Tests

.NET

Java, Python, Web,..

Free GUI Testing Tools / Capture & Replay Tools

Windows, MacOSX, Linux, iPhone, Android

.NET

Java

Linux, X Window

PowerShell

Python

Tcl/Tk

Web / HTML Testing in Python, Java, Javascript, PHP

Appium
Javascript Tools
  • Cypress, GitHub "cypress-io/cypress" - “Fast, easy and reliable testing for anything that runs in a browser”.
  • Galen Framework, GitHub "galenframework" - “Automated testing of look and feel for your responsive websites”. A Selenium-based Javascript framework.
  • GitHub.io "Karma - Spectacular Test Runner for Javascript", GitHub "Karma - The Spectacular Test Runner for JavaScript" - “Karma is essentially a tool which spawns a web server that executes source code against test code for each of the browsers connected”.
  • Nightwatch.js, GitHub "beatfactor/nightwatch".
    • “Nightwatch.js is an easy to use Node.js based End-to-End (E2E) testing solution for browser based apps and websites. It uses the powerful Selenium WebDriver API to perform commands and assertions on DOM elements”.
    • “Nightwatch.js works with cloud testing providers, such as SauceLabs and BrowserStack.
    • Twitter "Nightwatch.js".
    • Daniel und Kian "Continuous Deployment mit git, codeship.io und nightwatch.js bei youhear.it", 2014-06-14.
    • Installation:
      1. “sudo npm install nightwatch”.
      2. Install and run the Selenium Server - “java -jar selenium-server-standalone.2.42.0.jar”.
      3. “nightwatch” or “node nightwatch.js”
    • Configuration on Windows:
      • On Windows, as shipped, “nightwatch” aborts by the error message “ERROR There was an error while starting the test runner: Error: Failed to load external global file: External global file could not be located - using ./examples/globals.json”.
      • To fix this for Windows, please modifly these lines of the file ”./node_modules/nightwatch/examples/tests” of your local node module directory:
        "output_folder" : "./node_modules/nightwatch/examples/reports",
"custom_commands_path" : "./node_modules/nightwatch/examples/custom-commands",
"custom_assertions_path" : "",
"globals_path" : "./node_modules/nightwatch/examples/globals.json",
    • Configuration on Linux:
      • In case of a local installation of “nightwatch”, this “nightwatch.js” file worked for me:
        #!/usr/bin/env node
require('./node_modules/nightwatch/bin/runner.js');
    • General configuration tips:
      • You should build your own “nightwatch.json” in your local project directory, see official tutorial :-).
  • DalekJS, GitHub "DalekJS".
    • DalekJS "Get Started. This ain’t Selenium".
    • “Dalek's only dependency is node.js, there is no need to install Java or download the Selenium.jar file”.
    • Installation for a certain project directory. Though I was able to install DalekJS on Windows as total local installation, with Linux I must - as instructed by the documentation - dalek-cli as global, but dalekjs as local, else it does not appear with “dalek -v”. I decided to follow the instructions and installed the rest of the modules as local, too.
      1. sudo npm install -g dalek-cli

        . 

      2. npm install dalekjs --save-dev

        This creates the folder “node_modules” in your project directory, and installs some code there. 

      3. dalek -v

        for checking the proper installation. Both “dalek-cli” and “dalek” must be listed, so 2 modules, not just one. 

      4. dalek test/my_first_test.js
      5. sudo npm install dalek-browser-chrome --save-dev

        or 

        npm install dalek-browser-firefox --save-dev
      6. dalek test/my_first_test.js -b chrome

        or 

        dalek test/my_first_test.js -b firefox
        • There is the error on Windows ”» ERROR: dalek-driver-firefox: Binary not found: C:\Program Files\Mozilla Firefox\firefox.exe”, if Firefox is not installed at the default location.
        • There is the error on Windows ”» ERROR: dalek-browser-firefox: Error starting WebDriverServer, port 9006 in use”.
          • GitHub Issues for "dalekjs/dalek-browser-firefox" - “Firefox won't start on any port #11”.
          • If I set a larger timeout at “my_project\node_modules\dalek-browser-firefox\node_modules\portscanner\lib\portscanner.js”, I get the error “net.js:1236: throw new Error('Not running');”:
            • Original:
              • “var timeout = options.timeout || 400”
            • Changed to:
              • “var timeout = options.timeout || 4000”
      7. npm install -g dalek-reporter-html --save-dev
      8. mkdir report
      9. mkdir cd report
      10. mkdir dalek
      11. mkdir cd ..
      12. dalek test/my_first_test.js -r console,html

        This command expects that the folder “report/dalek” exists. If so, it creates the file “index.html” in that directory with all the information from your test run.

    • My personal experience of 2014-08: The current version of Dalek “hangs” if an assertion is not passed succesfully :-(
  • GitHub "puppeteer/puppeteer" - “Headless Chrome Node.js API”, “A Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium”.
  • Resources:
Katalon Studio
Playwright
Selenium
Selenium with Geb
Selenium with Codeception
SOAPSonar
SoupUI
Some other Tools
Online Services

Web Automation by direct HTTP Automation

Windows Macro Recorders

Basics
Tools

Windows Automation & Batch Systems

Macro Recorders in Desktop Applications

The Macro Recorder in OpenOffice

Just-Commercial GUI Testing Tools / Capture & Replay Tools

Web, Windows, WPF, HTML5, Flash, Flex, Silverlight, .NET, VCL and Java

Java & Web

  • QF-Test - “Das GUI Testtool für Java und Web”.

.NET

  • Telerik.
    • Telerik "Test Studio" ( formerly: “Telerik WebUI Test Studio” ) - “Functional Testing ( HTML, Silverlight, AJAX, WPF ), Load Testing, Performance Testing, Testing in Visual Studio, Explorative Testing, Manual Testing, Mobile Testing”.
  • Automated UI tests for WPF projects with Visual Studio.

Windows & Windows GUI

Windows Macro Recorders

Basics
Tools

Windows Automation & Batch Systems

Macro Recorders in Desktop Applications

The Macro Recorders in Microsoft Excel
The Macro Recorder in Microsoft Visual Studio
The Macro Recorder in Microsoft Word

Addons

OpenSource Software for End-to-End Testing of Databases

ORAYLIS BI.Quality

NBi

The Tool
Resources

Unit Test for Databases

Just-Commercial Software for Testing & Test Management

Just-Commercial Online Services for Testing & Test Management "In the Cloud"

Model-based Testing of Embedded Control Systems

  • The commercial PikeTec GmbH "TPT".
    • Tests.
      1. Test of MATLAB/Simulink models and TargetLink models test and verification of MATLAB Simulink models and TargetLink models.
      2. Test of ASCET models test and verification of ASCET models.
      3. Test of AUTOSAR software test and verification of AUTOSAR Software.
      4. Test of C-Code test and verification of C-Code, C-Programs.
      5. Test via CAN-Bus test and verification via CAN-Bus.
      6. Test via INCA und CANape test and verification via INCA and CANape.
      7. Test at HiL.
    • EN.Wikipedia "TPT (software)", DE.Wikipedia "TPT (Software)".

Resources

Test Anything Protocol ( TAP )

  • Test Anything Protocol ( TAP ).
    • “A simple text-based interface between testing modules in a test harness. It decouples the reporting of errors from the presentation of the reports. One of its major uses is for noise reduction; when you have a suite of many tests, making them TAP producers and using a TAP consumer to view them helps ensures that you will see everything you need to notice and diagnose breakage without being distracted by a flood of irrelevant success messages. It can assist other forms of analysis and statistics-gathering as well”.
    • “TAP started life as part of the test harness for Perl but now has implementations in C, C++, Python, PHP, Perl, Java, JavaScript, Go, Rust, and others. Consumers and producers do not have to be written in the same language to interoperate”.
  • Python.
  • Lua:
  • EN.Wikipedia "Test Anything Protocol".

TTCN-3

Commercial TTCN Testing Tools

Resources

YAML - YAML Ain't Markup Language, RAML, TAML,...

YAML Resources

RAML ( RESTful API Modeling Language )

YAML Software

Free YAML Online Services

TAML

Tools

Resources

Dynamic Code Analysis

Test Concept for Embedded Systems

Test Concept #1 - One-way Communication

Test Concept #1 - The classical "printf()"

  • Use “printf()” by an URART to a remote console on the host. In case of multithreading, multitasking this is sometimes the best quick + dirty solution.

Test Concept #1.2 - "Spy" Function of an RTOS

  • A “spy” function of an RTOS sends a binary, but ordered or tagged, data stream to the PC (via serial, USB, Ethernet, wireless, etc.) and the PC decodes it for capture and/or display. An advanced diagnostic tool might send this binary data over Ethernet to be captured by Wireshark and IIRC ( a custom filter within Wireshark ).

Test Concept #2 - Unit Test with Hardware Emulation

  1. Write a test for any module that will exercise the function at edge and corner cases.
  2. If the module code accesses hardware in the production version,
    1. Either supply adequiate hardware signals to the device under test ( Hardware in the Loop ( HIL ) ).
    2. And / or emulate the hardware and its input and output ( just use an if-query not to access hardware if the code is tested, bzt
      1. to use predefined static values.
      2. to use calculated values.
      3. to use values provided by a state machine.
      4. or values transfered from a host, which generates the values by a script, a calculation or with the help of a state machine.
  3. You might use test concept #1 for one-way communication.
  4. Somehow you have to start your tests, by uploading code which executes the tests instead of executing the application.

Test Concept #3 - Script Languages

Test Concept #3.1 - Script Language on the Target

Test Concept #3.1.1 - Dialog by a classical Monitor
  • Implement and use an on-target monitor ( e.g. “debug.com” or a monitor running in the firmware ) to transfer informations by UART to a remote console on a host.
Test Concept #3.1.2 - Application-specific Script Language on the Target
  1. Design a “command language” and a parser to run on the device and its application under test, so that the interface to the module functions can be triggered with various values.
  2. The input to that device under test could be scripted.

Test Concept #3.2 - Script Language on the Host

  1. Install a script language ( Python, Perl,..) which can work by serial communications.
  2. The target “opens” a new console on the host and executes script command by its own output.

Test Concept #3.3 - Script Language both on the Host and the target

  1. Combine concept #3.1.2 and #3.2 to let interact 2 applications by a common interface and interface language.

Some other Test Concepts

Blinded Experiment

  • EN.Wikipedia "Blinded experiment", DE.Wikipedia "Blindstudie" ( “Blindtest” ).
  • In German language for technical “Blinded Experiments”:
    • “Ein Blindversuch ist ein Vorversuch unter den gleichen Bedingungen wie das endgültige Experiment durchgeführt wird, ohne das das eigentliche Untersuchungs-Objekt hinzugenommen wird. Und wenn das erste Objekt den Vorversuch besteht, kann das eigentliche Experiment zu den gleichen Voraussetzungen gestartet und zum Abschluss gebracht werden. Wichtig: Es darf nicht das selbe Objekt sein, es müssen 2 ( gleiche ) Objekte sein: Das Test-Objekt und das Untersuchungs-Objekt. Der erste wird getestet und dann wird der gleiche Versuch mit dem zweiten Objekt durchgeführt”.

Magazines

Books

General

Books about Software Testing

Books about Code Review, Cross-Browser Testing ( and Load Testing )

Books about TTCN

Resources

GUI Testing

General

Objects for testing your own Web Automation skills

Implementations

Search for Resources

Resources

Windows Task Automation

Database Debugging

Testing Education & Certification Preparation

Chat / IRC Communities

Forums, Newsgroups, Communities


When this document changes ! Site Navigation ( My Business ! My Topics ! Imprint / Contact ! Privacy Policy ! Keyword Index ! ! Google+ Publisher "hemmerling" )

 
en/testing.html.txt · Last modified: 2025/01/15 19:57 (external edit) · []
Recent changes RSS feed Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki