timeoutLimitTest is incorrectly set when setInactivityLimit is used
lumpfish opened this issue · comments
#216 added setInactivityLimit as a parameter to the openjdk-systemtest LambdaLoadTest.
However, a bug in STF means that this has a side effect of preventing most of the tests in the test suite from running.
A typical test output is currently:
17:50:15 STF 17:50:14.928 - +------ Step 1 - Run lambda and stream load test
17:50:15 STF 17:50:14.928 - | Run foreground process
17:50:15 STF 17:50:14.928 - | Program: /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdkbinary/j2sdk-image/bin/java
17:50:15 STF 17:50:14.928 - | Mnemonic: LT
17:50:15 STF 17:50:14.928 - | Echo: ECHO_ON
17:50:15 STF 17:50:14.928 - | Expectation: CLEAN_RUN within 1h
17:50:15 STF 17:50:14.928 - |
17:50:15 STF 17:50:14.928 - Running command: /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdkbinary/j2sdk-image/bin/java -Xgcpolicy:gencon -Xgc:concurrentScavenge -classpath /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/jvmtest/system/stf/stf.load/bin:/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/jvmtest/system/stf/stf.core/bin:/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/jvmtest/system/systemtest_prereqs/log4j-2.13.3/log4j-api-2.13.3.jar:/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/jvmtest/system/systemtest_prereqs/log4j-2.13.3/log4j-core-2.13.3.jar:/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/jvmtest/system/systemtest_prereqs/junit-4.12/junit-4.12.jar:/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/jvmtest/system/systemtest_prereqs/junit-4.12/hamcrest-core-1.3.jar:/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/jvmtest/system/openjdk-systemtest/openjdk.test.lambdasAndStreams/bin net.adoptopenjdk.loadTest.LoadTest -resultsDir /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdk-tests/TKG/test_output_16058941429755/LambdaLoadTest_ConcurrentScavenge_0/20201120-175013-LambdaLoadTest/results -resultsPrefix 1.LT. -inactivityLimit 60m -reportFailureLimit 1 -abortAtFailureLimit 10 -maxTotalLogFileSpace 200M -maxSingleLogSize 1/25 -suite.lambda.threadCount 2 -suite.lambda.inventoryFile /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdk-tests/TKG/test_output_16058941429755/LambdaLoadTest_ConcurrentScavenge_0/20201120-175013-LambdaLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/lambda.xml -suite.lambda.inventoryExcludeFile none -suite.lambda.totalNumberTests 200 -suite.lambda.selection random -suite.lambda.seed -1 -suite.lambda.repeatCount 1 -suite.lambda.thinkingTime 0ms..0ms
17:50:15 STF 17:50:14.928 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdk-tests/TKG/test_output_16058941429755/LambdaLoadTest_ConcurrentScavenge_0/20201120-175013-LambdaLoadTest/results/1.LT.stderr
17:50:15 STF 17:50:14.928 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdk-tests/TKG/test_output_16058941429755/LambdaLoadTest_ConcurrentScavenge_0/20201120-175013-LambdaLoadTest/results/1.LT.stdout
17:50:15 STF 17:50:14.933 - Monitoring processes: LT
17:50:15 LT 17:50:15.548 - Load test parameters
17:50:15 LT 17:50:15.552 - Time limited = true
17:50:15 LT 17:50:15.552 - Time limit = null
17:50:15 LT 17:50:15.552 - abortIfOutOfMemory = true
17:50:15 LT 17:50:15.552 - reportFailureLimit = 1
17:50:15 LT 17:50:15.552 - abortAtFailureLimit = 10
17:50:15 LT 17:50:15.552 - maxTotalLogFileSpace = 209715200
17:50:15 LT 17:50:15.552 - maxSingleLogSize = 8388608
17:50:15 LT 17:50:15.552 - Parameters for suite 0
17:50:15 LT 17:50:15.552 - Suite name = lambda
17:50:15 LT 17:50:15.552 - Number threads = 2
17:50:15 LT 17:50:15.552 - Supplied seed = -1
17:50:15 LT 17:50:15.553 - Inventory file = /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdk-tests/TKG/test_output_16058941429755/LambdaLoadTest_ConcurrentScavenge_0/20201120-175013-LambdaLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/lambda.xml
17:50:15 LT 17:50:15.553 - Exclude file = none
17:50:15 LT 17:50:15.553 - Number tests = 200
17:50:15 LT 17:50:15.553 - Repeat count = 1
17:50:15 LT 17:50:15.553 - Thinking time = 0ms..0ms
17:50:15 LT 17:50:15.553 - Selection mode = random
17:50:15 LT 17:50:15.553 - Actual seed = 1605434495915
17:50:15 LT 17:50:15.557 - Parsing inventory file. Root=/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdk-tests/TKG/test_output_16058941429755/LambdaLoadTest_ConcurrentScavenge_0/20201120-175013-LambdaLoadTest/results/1.LT.inventory File=openjdk.test.load/config/inventories/lambdasAndStreams/lambda.xml
17:50:15 LT 17:50:15.570 - Final test list:
17:50:15 LT 17:50:15.570 - 0 JUnit[net.adoptopenjdk.test.lambda.ClassLibraryLambdaTests] Weighting=1
17:50:15 LT 17:50:15.570 - 1 JUnit[net.adoptopenjdk.test.lambda.TestLambdaCapture] Weighting=1
17:50:15 LT 17:50:15.570 - 2 JUnit[net.adoptopenjdk.test.lambda.TestLambdaContexts] Weighting=1
17:50:15 LT 17:50:15.570 - 3 JUnit[net.adoptopenjdk.test.lambda.TestLambdaDefaultsAndStatics] Weighting=1
17:50:15 LT 17:50:15.570 - 4 JUnit[net.adoptopenjdk.test.lambda.TestLambdaInferredTyping] Weighting=1
17:50:15 LT 17:50:15.570 - 5 JUnit[net.adoptopenjdk.test.lambda.TestLambdaJavaInterfaces] Weighting=1
17:50:15 LT 17:50:15.571 - 6 JUnit[net.adoptopenjdk.test.lambda.TestLambdaLexicalScoping] Weighting=1
17:50:15 LT 17:50:15.571 - 7 JUnit[net.adoptopenjdk.test.lambda.TestLambdaMethodReferences] Weighting=1
17:50:15 LT 17:50:15.571 - 8 JUnit[net.adoptopenjdk.test.lambda.TestLambdaMultithreaded] Weighting=1
17:50:15 LT 17:50:15.571 - 9 JUnit[net.adoptopenjdk.test.lambda.TestLambdaNested] Weighting=1
17:50:15 LT 17:50:15.571 - 10 JUnit[net.adoptopenjdk.test.lambda.TestLambdaRecursive] Weighting=1
17:50:15 LT 17:50:15.571 - 11 JUnit[net.adoptopenjdk.test.lambda.TestLambdasUserDefinedInterface] Weighting=1
17:50:15 LT 17:50:15.571 - 12 JUnit[net.adoptopenjdk.test.streams.TestParallelStreamOperations] Weighting=1
17:50:15 LT 17:50:15.571 - 13 JUnit[net.adoptopenjdk.test.streams.TestStreamOperations] Weighting=1
17:50:15 LT 17:50:15.609 - Starting thread. Suite=0 thread=0
17:50:15 LT 17:50:15.610 - Starting thread. Suite=0 thread=1
17:50:16 LT 17:50:15.768 - Thread completed. Suite=0 thread=1
17:50:16 LT 17:50:15.781 - Thread completed. Suite=0 thread=0
17:50:16 LT 17:50:15.810 - Test stopped due to reaching runtime limit
17:50:16 LT 17:50:15.810 - Load test completed
17:50:16 LT 17:50:15.811 - Ran : 2
17:50:16 LT 17:50:15.811 - Passed : 2
17:50:16 LT 17:50:15.811 - Failed : 0
17:50:16 LT 17:50:15.811 - Result : PASSED
17:50:16 STF 17:50:15.826 - Monitoring Report Summary:
17:50:16 STF 17:50:15.826 - o Process LT ended with the expected exit code (0)
17:50:16 STF 17:50:15.826 - EXECUTE stage completed
17:50:16 STF 17:50:15.832 -
17:50:16 STF 17:50:15.832 - ==================== T E A R D O W N ====================
17:50:16 STF 17:50:15.832 - Running teardown: perl /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_s390x_linux/openjdk-tests/TKG/../TKG/test_output_16058941429755/LambdaLoadTest_ConcurrentScavenge_0/20201120-175013-LambdaLoadTest/tearDown.pl
17:50:16 STF 17:50:15.864 - TEARDOWN stage completed
17:50:16 STF 17:50:15.870 -
17:50:16 STF 17:50:15.870 - ===================== R E S U L T S =====================
17:50:16 STF 17:50:15.870 - Stage results:
17:50:16 STF 17:50:15.870 - setUp: pass
17:50:16 STF 17:50:15.870 - execute: pass
17:50:16 STF 17:50:15.870 - teardown: pass
17:50:16 STF 17:50:15.870 -
17:50:16 STF 17:50:15.870 - Overall result: PASSED
17:50:16
17:50:16 LambdaLoadTest_ConcurrentScavenge_0_PASSED
Note the following lines in the output:
17:50:15 LT 17:50:15.552 - Time limited = true
17:50:15 LT 17:50:15.552 - Time limit = null
......
17:50:16 LT 17:50:15.810 - Test stopped due to reaching runtime limit
When inactivityLimit is specified, STF is incorrectly setting timeLimitedTest to true. This in turn means that once the test starts its elapsed time is compared with the timeLimit of null which is interpreted as being exceeded and the test in terminated.
The test output should really look something like:
LT 15:26:34.103 - Starting thread. Suite=0 thread=0
LT 15:26:34.104 - Starting thread. Suite=0 thread=1
LT 15:26:54.154 - Completed 3.5%. Number of tests started=7
LT 15:27:14.183 - Completed 3.5%. Number of tests started=7 (+0)
LT 15:27:34.187 - Completed 3.5%. Number of tests started=7 (+0)
LT 15:27:54.151 - Completed 30.0%. Number of tests started=60 (+53)
LT 15:28:14.102 - Completed 30.0%. Number of tests started=60 (+0)
LT 15:28:34.106 - Completed 30.0%. Number of tests started=60 (+0)
LT 15:28:54.125 - Completed 54.5%. Number of tests started=109 (+49)
LT 15:29:14.190 - Completed 54.5%. Number of tests started=109 (+0)
LT 15:29:34.142 - Completed 54.5%. Number of tests started=109 (+0)
LT 15:29:54.099 - Completed 67.5%. Number of tests started=135 (+26)
LT 15:30:14.122 - Completed 67.5%. Number of tests started=135 (+0)
LT 15:30:34.097 - Completed 67.5%. Number of tests started=135 (+0)
LT 15:30:54.165 - Completed 78.0%. Number of tests started=156 (+21)
LT 15:31:14.100 - Completed 78.0%. Number of tests started=156 (+0)
STF 15:31:32.135 - Heartbeat: Process LT is still running
LT 15:31:34.095 - Completed 78.0%. Number of tests started=156 (+0)
LT 15:31:54.128 - Completed 84.0%. Number of tests started=168 (+12)
LT 15:32:14.188 - Completed 84.0%. Number of tests started=168 (+0)
LT 15:32:31.557 - Thread completed. Suite=0 thread=0
LT 15:32:34.098 - Completed 100.0%. Number of tests started=200 (+32)
LT 15:32:54.189 - Completed 100.0%. Number of tests started=200 (+0)
LT 15:32:56.642 - Thread completed. Suite=0 thread=1
LT 15:32:56.699 - Load test completed
LT 15:32:56.699 - Ran : 200
LT 15:32:56.699 - Passed : 200
LT 15:32:56.699 - Failed : 0
LT 15:32:56.699 - Result : PASSED
STF 15:32:56.762 - Monitoring Report Summary:
STF 15:32:56.762 - o Process LT ended with the expected exit code (0)
STF 15:32:56.762 - EXECUTE stage completed
STF 15:32:57.190 -
When this is fixed the LambdaLoadTests will start executing correctly again, so all releases and platforms should be tested with the PR before it is merged to check there are no regressions.
Fixed by #102