Extend configuration files to support individual test configurations

[Matthias Simon]

We need a way to configure test runs individually, because:

• some tests may require higher time out limits.
• or performance and capacity tests require a special preset of values (i.e. numberOfUEs)
• or some tests need to be run multiple times with different configuration (i.e. testConn(useIPv6 := true) and testConn(useIPv4 := false)
• ...

Problem is, the parameters file is not integrated in ntt and package.yml provides only rudimentary configuration. Therefore we should:

• formalize and integrate the parameters file into existing ntt tooling (ntt show --json).
• and align package.yml by extending its runtime configuration capabilities.

YAML Scheme

This scheme will be used for parameters files and for extending package.yml, as well.

Test Configuration

The basic building block is a test configuration object. A test configuration should include:

• a full qualified testcase identifier
• a set of testcase parameters
• a set of module parameters
• a timeout limit
• a condition when the configuration is active/deactive.

Suggested scheme:

# test is a string specifying a full qualified testcase identifier. Testcase parameters are optional.
# If testcase parameters are omitted it's in the execution engine's responsibility to parameterize testcases properly.
test: "test.FeatureA(23, 5)"

# timeout species the timeout limit in seconds.
timeout: 23.5

# parameters maps full qualified module parameters to their values.
# The values should be stringable and will be parsed into TTCN-3 values during execution.
parameters:
    "Mod1.ParamA": 23
    "Mod2.ParamB": "{ x := 23, y := 5}"

# only describes when a test should be run. This example means: test.FeatureA should
# only be run when preset1 is active.
only:
    presets:
        - "preset1"

# except describes when a test should _not_ run.
except:
    presets:
        - "preset2"
        - "preset3"

Parameters File

The parameters file contains a single test configuration as root object. This root object is extended by a set of alternate test configurations (== presets) and a list of test specific configurations.

Suggested scheme:

# Global configuration, to be used as default for all test runs.
timeout: 10.0
parameters:
    Mod1.ParamA: 5

# Presets specify alternative global configurations (type presets: map[string]TestConfiguration)
presets:
    "capacity":
        timeout: 300.0
        parameters:
            "Config.numberOfUEs": 10000
    "performance":
        timeout: 5.0

# Execute specifies a list of test specific configurations (type execute: []TestConfiguration)
execute:
    # Run test with current global configuration. Parentheses are optional.
    - test: "test.FeatureA"
    
    # Run test with individual timeout of 2 second.
    - test: "test.FeatureA()"
      timeout: 2.0

    #  Run test with actual test case parameters, only for default runs.
    - test: "test.FeatureA(23, 5)"
      except:
          presets:
              - "preset1"

Behaviour

Overriding

• presets override global configuration
• test specific configurations override presets

Example:

timeout: 1.0
parameters:
    A: "global"
    B: "global"

preset:
    p:
         timeout: 2.0
         parameters:
              B: "preset"
              C: "preset"

execute:
    - test: "tc1"
      timeout: 3.0
      parameters:
          C: "test"

When preset "p" is active, test tc1 will be executed with this configuration:

timeout: 3.0
parameters:
    A: "global"
    B: "preset"
    C: "test"

Execution

Every entry in execute behaves like a individual test execution.
Example: Assume a test suite with three tests (Foo, Bar and Baz) and following parameters file:

execute:
   - test: "test.Foo"
   - test: "test.Foo(2)"
   - test: "test.Bar()"

• Scenario 1: User selects whole test suite for execution.
  Result: 4 runs in total: Foo will be run twice (once with default parameters and once with parameter 2), Bar and Baz once.
• Scenario 2: User select test 'Baz' for execution.
  Result: 1 run in total: Only Baz is executed with global parameters, even though it is not explicitly listed.
• Scenario 3: User selects test 'Bar' for execution.
  Result: 1 run in total: Only Bar is executed with test specific parameters.
• Scenario 4: User selects test 'Foo' for execution.
  Result: 2 runs in total: Foo will be run twice (once with default testcase parameters and once with parameter 2)

Open Questions

• Should we allow variable substitution for values like in the rest of package.yml (ie "Mod1.DataDir": "${SCT_SOURCE_DIR}/share")?
• Should we allow patterns/regex in except:presets, only:presets and test (ie test: "test.Feature*")?
• Should we AND or OR multiple entries in except:presets and only:presets?
• What is the expected behavior when the global configuration contains a test or except spec?
• What is the expected behavior when a preset contains a test or except spec?
• What is the expected behavior when the test specific configuration does not contain a test spec?

[Matthias Simon]

• implement initial data structures
• integrate with ntt show --json
• replace Manifest.timeout with TestConfiguration
• check TTCN-3 values for errors
• implement variable substitution
• add test-globs

[moosq]

Testcase parameters are optional.

If no Testcase parameters are present in the ttcn-3 definition () can be omitted. Default values for testcase parameters are currently not supported

[moosq]

execute:

• test: "test.Foo"
• test: "test.Foo(2)"
• test: "test.Bar()"

• remark: (Scenario1) Baz is running (with default configuration) even though it is not listed
• default parameters are currently not supported

[Matthias Simon]

Bug or Feature?

Conflicting map-keys produce an error. Example:

# ./package.yml
parameters_file: ./example.parameters
parameters:
    "Foo": 5

# ./example.parameters
parameters:
    "Foo": 23

$ ntt show --json
example.parameters: line 2: key "Foo" already set in map