Table of Contents

[hemmerling] Testing

Organizations

Fairs, Conferences, Events

Certification

Testing Experts

Crowdtesting Jobs

Basics

  • Explorative testing = Manual testing, with testplan.

Questions about the Testing of Embedded Software

  1. How to test embedded software ? Under the conditions
    1. Most of the code is machine-dependent, i.e. a function to be tested accesses hardware ( reads and writes registers, while the results depend on external conditions and events ).
    2. Resources ( RAM,.. ) are limited, in comparison to the well-known standard of deskop PCs.
    3. Use of standard ( free, OpenSource, commercial )Test frameworks and especially Unit Test frameworks.

Mockup - Simulation of Objects

  • See [unittest.html|Unit Test / Module Test]].

Keyword-based Testing

Property-based Testing

Scala ( and Java )

Haskell

F#

Javascript

Instrumentation Frameworks for Dynamic Application Testing

Test Mangement, Functional Test Tools, Automation Frameworks, GUI Testing Tools / Capture & Replay Tools

Important Dynamic Code Analysis Tools, according to VDC Research in 2015, 2016-08

  • RapiTime/RapiCover (Rapita Systems).
  • LDRA Testbed (LDRA).
  • SilkTest (Borland / Micro Focus).
  • HP.
    • HP Quality Center / ALM.
    • HP Security Fortify (HP).
  • Rational Test RealTime/Tau Tester (IBM Rational).
  • TestQuest (BSQUARE).
  • Cantata++ /AdaTEST (QA Systems).
  • Test Center (Keynote/DeviceAnywhere).
  • Parasoft C++test/Insure++/Standard Compliance (Parasoft).
  • In-house developed tool.
  • MobileCloud (Perfecto Mobile).
  • Coverity Quality Adviser (Coverity).
  • VectorCAST (Vector Software).
  • Veracode (Veracode), 2016-08
  • SW-Assure (Monroe Software), 2016-08.

Some other General Test Management Tools

Testdata Creation and Testdata Management

  • The commercial testdata management system Averna Proligent.
  • The OpenSource GenerateData.com - Open-source data generator script.
  • The OpenSource Damhave Systems "sequence" - “A UNIX/Linux command line tool for generation binary or ASCII test sequences. Sequence is specially useful in bash scripts for test data generation when testing and debugging your own linux device drivers or playing with flash or MTD devices”.

Integration & Automation Frameworks

.NET

Java, Python, Web,..

Free GUI Testing Tools / Capture & Replay Tools

Windows, MacOSX, Linux, iPhone, Android

.NET

Java

Linux, X Window

PowerShell

Python

Tcl/Tk

Web / HTML Testing in Python, Java, Javascript, PHP

Appium
Selenium
Selenium with Geb
Selenium with Codeception
SOAPSonar
SoupUI
Javascript Tools
  • Galen Framework, GitHub "galenframework" - “Automated testing of look and feel for your responsive websites”. A Selenium-based Javascript framework.
    • “Nightwatch.js is an easy to use Node.js based End-to-End (E2E) testing solution for browser based apps and websites. It uses the powerful Selenium WebDriver API to perform commands and assertions on DOM elements”.
    • “Nightwatch.js works with cloud testing providers, such as SauceLabs and BrowserStack.
    • Installation:
      1. “sudo npm install nightwatch”.
      2. Install and run the Selenium Server - “java -jar selenium-server-standalone.2.42.0.jar”.
      3. “nightwatch” or “node nightwatch.js”
    • Configuration on Windows:
      • On Windows, as shipped, “nightwatch” aborts by the error message “ERROR There was an error while starting the test runner: Error: Failed to load external global file: External global file could not be located - using ./examples/globals.json”.
      • To fix this for Windows, please modifly these lines of the file ”./node_modules/nightwatch/examples/tests” of your local node module directory:
        "output_folder" : "./node_modules/nightwatch/examples/reports",
        "custom_commands_path" : "./node_modules/nightwatch/examples/custom-commands",
        "custom_assertions_path" : "",
        "globals_path" : "./node_modules/nightwatch/examples/globals.json",
    • Configuration on Linux:
      • In case of a local installation of “nightwatch”, this “nightwatch.js” file worked for me:
        #!/usr/bin/env node
        require('./node_modules/nightwatch/bin/runner.js');
    • General configuration tips:
      • You should build your own “nightwatch.json” in your local project directory, see official tutorial :-).
    • “Dalek's only dependency is node.js, there is no need to install Java or download the Selenium.jar file”.
    • Installation for a certain project directory. Though I was able to install DalekJS on Windows as total local installation, with Linux I must - as instructed by the documentation - dalek-cli as global, but dalekjs as local, else it does not appear with “dalek -v”. I decided to follow the instructions and installed the rest of the modules as local, too.
      1. sudo npm install -g dalek-cli

        .

      2. npm install dalekjs --save-dev

        This creates the folder “node_modules” in your project directory, and installs some code there.

      3. dalek -v

        for checking the proper installation. Both “dalek-cli” and “dalek” must be listed, so 2 modules, not just one.

      4. dalek test/my_first_test.js
      5. sudo npm install dalek-browser-chrome --save-dev

        or

        npm install dalek-browser-firefox --save-dev
      6. dalek test/my_first_test.js -b chrome

        or

        dalek test/my_first_test.js -b firefox
        • There is the error on Windows ”» ERROR: dalek-driver-firefox: Binary not found: C:\Program Files\Mozilla Firefox\firefox.exe”, if Firefox is not installed at the default location.
        • There is the error on Windows ”» ERROR: dalek-browser-firefox: Error starting WebDriverServer, port 9006 in use”.
          • GitHub Issues for "dalekjs/dalek-browser-firefox" - “Firefox won't start on any port #11”.
          • If I set a larger timeout at “my_project\node_modules\dalek-browser-firefox\node_modules\portscanner\lib\portscanner.js”, I get the error “net.js:1236: throw new Error('Not running');”:
            • Original:
              • “var timeout = options.timeout || 400”
            • Changed to:
              • “var timeout = options.timeout || 4000”
      7. npm install -g dalek-reporter-html --save-dev
      8. mkdir report
      9. mkdir cd report
      10. mkdir dalek
      11. mkdir cd ..
      12. dalek test/my_first_test.js -r console,html

        This command expects that the folder “report/dalek” exists. If so, it creates the file “index.html” in that directory with all the information from your test run.

    • My personal experience of 2014-08: The current version of Dalek “hangs” if an assertion is not passed succesfully :-(
Some other Tools
Online Services

Web Automation by direct HTTP Automation

Windows Macro Recorders

Basics
Tools

Windows Automation & Batch Systems

Macro Recorders in Desktop Applications

The Macro Recorder in OpenOffice

Just-Commercial GUI Testing Tools / Capture & Replay Tools

Web, Windows, WPF, HTML5, Flash, Flex, Silverlight, .NET, VCL and Java

Java & Web

  • QF-Test - “Das GUI Testtool für Java und Web”.

.NET

    • Telerik "Test Studio" ( formerly: “Telerik WebUI Test Studio” ) - “Functional Testing ( HTML, Silverlight, AJAX, WPF ), Load Testing, Performance Testing, Testing in Visual Studio, Explorative Testing, Manual Testing, Mobile Testing”.
  • Automated UI tests for WPF projects with Visual Studio.

Windows & Windows GUI

Windows Macro Recorders

Basics
Tools

Windows Automation & Batch Systems

Macro Recorders in Desktop Applications

The Macro Recorders in Microsoft Excel
The Macro Recorder in Microsoft Visual Studio
The Macro Recorder in Microsoft Word

Addons

OpenSource Software for End-to-End Testing of Databases

ORAYLIS BI.Quality

NBi

The Tool
Resources

Unit Test for Databases

Just-Commercial Software for Testing & Test Management

Just-Commercial Online Services for Testing & Test Management "In the Cloud"

Model-based Testing of Embedded Control Systems

  • The commercial PikeTec GmbH "TPT".
    • Tests.
      1. Test of MATLAB/Simulink models and TargetLink models test and verification of MATLAB Simulink models and TargetLink models.
      2. Test of ASCET models test and verification of ASCET models.
      3. Test of AUTOSAR software test and verification of AUTOSAR Software.
      4. Test of C-Code test and verification of C-Code, C-Programs.
      5. Test via CAN-Bus test and verification via CAN-Bus.
      6. Test via INCA und CANape test and verification via INCA and CANape.
      7. Test at HiL.

Resources

TTCN-3

Commercial TTCN Testing Tools

Resources

YAML - YAML Ain't Markup Language

YAML Resources

RAML ( RESTful API Modeling Language )

YAML Software

Free YAML Online Services

Dynamic Code Analysis

Test Concept for Embedded Systems

Test Concept #1 - One-way Communication

Test Concept #1 - The classical "printf()"

  • Use “printf()” by an URART to a remote console on the host. In case of multithreading, multitasking this is sometimes the best quick + dirty solution.

Test Concept #1.2 - "Spy" Function of an RTOS

  • A “spy” function of an RTOS sends a binary, but ordered or tagged, data stream to the PC (via serial, USB, Ethernet, wireless, etc.) and the PC decodes it for capture and/or display. An advanced diagnostic tool might send this binary data over Ethernet to be captured by Wireshark and IIRC ( a custom filter within Wireshark ).

Test Concept #2 - Unit Test with Hardware Emulation

  1. Write a test for any module that will exercise the function at edge and corner cases.
  2. If the module code accesses hardware in the production version,
    1. Either supply adequiate hardware signals to the device under test ( Hardware in the Loop ( HIL ) ).
    2. And / or emulate the hardware and its input and output ( just use an if-query not to access hardware if the code is tested, bzt
      1. to use predefined static values.
      2. to use calculated values.
      3. to use values provided by a state machine.
      4. or values transfered from a host, which generates the values by a script, a calculation or with the help of a state machine.
  3. You might use test concept #1 for one-way communication.
  4. Somehow you have to start your tests, by uploading code which executes the tests instead of executing the application.

Test Concept #3 - Script Languages

Test Concept #3.1 - Script Language on the Target

Test Concept #3.1.1 - Dialog by a classical Monitor
  • Implement and use an on-target monitor ( e.g. “debug.com” or a monitor running in the firmware ) to transfer informations by UART to a remote console on a host.
Test Concept #3.1.2 - Application-specific Script Language on the Target
  1. Design a “command language” and a parser to run on the device and its application under test, so that the interface to the module functions can be triggered with various values.
  2. The input to that device under test could be scripted.

Test Concept #3.2 - Script Language on the Host

  1. Install a script language ( Python, Perl,..) which can work by serial communications.
  2. The target “opens” a new console on the host and executes script command by its own output.

Test Concept #3.3 - Script Language both on the Host and the target

  1. Combine concept #3.1.2 and #3.2 to let interact 2 applications by a common interface and interface language.

Magazines

Books

General

Books about Software Testing

Books about Code Review, Cross-Browser Testing ( and Load Testing )

Books about TTCN

Resources

GUI Testing

General

Objects for testing your own Web Automation skills

Implementations

Search for Resources

Resources

Windows Task Automation

Database Debugging

Testing Education & Certification Preparation

Chat / IRC Communities

Forums, Newsgroups, Communities


When this document changes ! Site Navigation ( My Business ! My Topics ! Imprint / Contact ! Privacy Policy ! Keyword Index ! ! Google+ Publisher "hemmerling" )

 
en/testing.html.txt · Last modified: 2017/08/13 17:36 (external edit) · []
Recent changes RSS feed Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki