hastern / nightmare

nightmare is a tool for automatic testing of non-interactive commandline applications.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

NIGHTMARE

Nightmare Is of Generous Help when Testing; May Arnold be Remembered Eternally

You must really, really love to test!


nightmare is a tool for automatic testing of non-interactive commandline applications.

usage: nightmare-3.0.0-py3.7.egg [-h] [--bench BENCH] [--suite SUITE]
                                 [--dut DUT] [--test TEST [TEST ...]]
                                 [--timeout TIMEOUT] [--arnold] [--save FILE]
                                 [--limit LIMIT] [--quiet] [--verbose]
                                 [--commands] [--length] [--info-only]
                                 [--pipe-streams] [--output-fails]
                                 [--unify-fails] [--no-color] [--continue]
                                 [--error] [--ignoreEmptyLines] [--relative]
                                 [--cr] [--ln] [--crln] [--gui] [--no-gui]
                                 [--version]

A test tool for non-interactive command line programs

optional arguments:
  -h, --help            show this help message and exit
  --gui                 Use the GUI (experimental and unstable).
  --no-gui              Don't use the GUI.
  --version             Display version information

Test selection:
  --bench BENCH         File which contains the testbench.
  --suite SUITE         Use testsuite SUITE from the testbench.
  --dut DUT, --DUT DUT  Set the device under test.
  --test TEST [TEST ...]
                        Run only the specified tests
  --timeout TIMEOUT     Set a global timeout for all tests.
  --arnold, -a          Use the arnold mode (requires pyparsing module)
  --save FILE           Save the testsuite as FILE

Output Control:
  --limit LIMIT         Set a (soft) limit for a number of Bytes, after which
                        output piping will we stopped. Checks are made after
                        each line.
  --quiet, -q           Quiet mode. There will be no output except results.
  --verbose, -v         Verbose mode. The program gets chatty (default).
  --commands, -C        Show the command executed for each test.
  --length, -l          Print only the number of tests in the suite.
  --info-only, -i       Display only test information, but don't run them.
  --pipe-streams, -p    Redirect DUT output to their respective streams.
  --output-fails, -o    Redirect DUT output from failed tests to their
                        respective streams.
  --unify-fails, -u     Display the unified diff of output and expectation.
  --no-color            Don't use any colored output.

Test Flow:
  --continue, -c        Continuous mode (Don't halt on failed tests).
  --error, -e           Same as '-c', but will halt if an error occurs.
  --ignoreEmptyLines, -L
                        Ignore empty lines
  --relative, -r        Use a path relative to the testbench path.
  --cr                  Force the line separation character (Mac OS).
  --ln                  Force the line separation character (Unix / Mac OS-X).
  --crln                Force the line separation character (Windows).

Additional to the commandline interface there is a GUI using the wxPython. The GUI is limited in its capabilities compared to the CLI, especially when it comes to testbench editing.

Be careful when you save your testbench! You might loose data, if it originates from a handwritten testbench.

Testbench files

A collection of test is called a "testbench". Inside a testbench there are a number of suites grouping tests together.

The testbench files a normal python files, which get evaluated by calling the internal "execfile" function. This might not be the safest approach, but it is definitely one of the easiest ones.

Here is a example for a minimal testbench.

#!/usr/bin/env python

# pyTest - Testsuite
# Saved at 21:12:26
#

# Device Under Test
DUT = "echo"

# Test definitions
suite = [
    Test(
        name = "Test 1",
        description = "A successfull run",
        command = "$DUT pyTest",
        stdout = "pyTest",
        returnCode = 0
    ),
    Test(
        name = "Test 2",
        description = "Testing output on stderr",
        command = "$DUT pyTest 1>&2",
        stderr = "pyTest"
    )
]

This example contains on testsuite name "suite" with two tests. I hope the syntax is selfexplanatory.

The Pattern $DUT is a substitute for the application that gets tested.

Here's a list of all possible fields in a test definition:

  • name: The (brief) name of the test.
  • description: A longer description, about the purpose of this test.
  • command: The command to be executed.
  • stdout: The expected result on stdout.
  • stderr: The expected result on stderr.
  • returnCode: The expected returncode.
  • timeout: Time in seconds before the process gets automatically killed.

Almost every field in the test is optional expect the command. The reason should be obvious.

One reason for the testfiles to be real python script is the possibility to use lambda functions for testing the output against an expectation.

The following example shows a test using a lambda function:

Test(
    name "Lambda",
    description = "A test with lambda function",
    command = "echo Hello World",
    stdout = lambda x : x.find("o") > 0
)

All expectation fields (stdout, stderr, returnCode) may contain lambda a lambda expression.

In addition it is possible to use regular expression as expectation. This is done by a simple "regex:"-prefix followed by the regular expression.

The following example shows a test using a regular expression:

Test(
    name "Regex",
    description = "A test with regular expression",
    command = "echo Hello World",
    stdout = "regex:^[a-zA-Z ]+$"
)

In addition nightmare supports so called Expecation-objects as value for stdout and stderr. These classes must have a __call__ method with a single argument, which is the output of the corresponding stream. The method should compare the output with the expectation and return a boolean value.

Another special type are so called Stringifier-objects. They also must implement a __call__ method, but their result is a 2-tuple containing 2 lists of strings, that are passed onto nightmare's internal diff (-u) tool.

There are three predefined Expectations/ Stringifiers:

  • ExpectFile(filename): compares the output byte-wise with a given file.

      Test(
          ...
          stdout = ExpectFile("output.txt")
      )
    
  • Stringifier(object): compares the output with the string-representation of the given object.

      # Assumption: there exists a class Image
      img = Image.read("image.png")
    
      Test(
          ...
          stdout = Stringifier(img)
      )
    
  • StringifiedFile(filename): loads the contents of a text-file for line by line comparison.

      Test(
          ...
          stdout = StringifiedFile("output.txt")
      )
    

To make the test benches more readable the following expectations are defined.

  • Regex: Syntactic sugar Expectation to represent a regular expression.

      Test(
          name="stdout Regular Expression",
          description="Test if the comparison with regular expressions works correctly",
          command="echo this should be a success",
          stdout=Regex("^this [a-z]+ [a-z ]+s.c*.s*$"),
      )
    
  • Contains: Checks that a list of strings is in the output

      Test(
          name="Example 4",
          description="Contains-Expectation fail",
          command="echo Some more text",
          stdout=Contains("text")
      )
    
  • ContainsNot: Checks that a list of strings is not in the output

      Test(
          name="Example 4",
          description="Contains-Expectation fail",
          command="echo Some more text",
          stdout=ContainsNot("test")
      )
    
  • Startswith: Syntactic sugar Expectation for finding multiple texts in the output

      Test(
          name="Example 9",
          description="Syntactic sugar: Startswith-Expectation success",
          command="$DUT Some more text",
          stdout=Startswith("Some more"),
      )
    
  • NonZero: Syntactic sugar Expectation for return code checking

  • Negative: Syntactic sugar Expectation for return code checking

History / Background

The development started in 2012 at FH-Wedel as a addition / replacement to the aging "arnold"-tool, which is a tcl-script. Since I'm a notorious windows user I was annoyed by the fact, that tcl/expect barely works on windows. Being a fan of the python language I decided to develop a new tool which fulfills the same requirements but by using a (in my humble opinion) more modern language.

As part of my job at the FH Wedel it was successfully used as the primary testing tool, to check if the implementation of the programming exercise meet the required specifications. Since "arnold" is an acronym, this tool's name needed to be equally ridiculous. The name "nightmare" was given as a consequence, because most of the students in the exercises absolutely hated the strictness of my tests.

About

nightmare is a tool for automatic testing of non-interactive commandline applications.

License:Other


Languages

Language:Python 98.3%Language:Makefile 1.5%Language:C 0.2%