RC3: tomcat validation error only on default
heshpdx opened this issue · comments
On release candidate 3, running MSJDK-17 on aarch64. Default size does not validate, but the small size and the large size are both ok.
Default:
$ ../JAVA/jdk-17.0.8.1+1/bin/java -jar dacapo-23.9-RC3-chopin.jar -v -n 3 tomcat
Class name: org.dacapo.harness.Tomcat
Configurations:
short Tomcat servlet container
long Runs the tomcat sample web applications
author Apache Software Foundation
license Apache License, Version 2.0
copyright Copyright (c) Apache Software Foundation
url http://tomcat.apache.org
version tomcat 10.1.11
sizes default large small
Using scaled threading model. 160 processors detected, 160 threads used to drive the workload, in a possible range of [1,512]
Version: tomcat 10.1.11 (use -p to print nominal benchmark stats)
Server thread created
===== DaCapo 23.9-RC3-chopin tomcat starting warmup 1 =====
Loading web application
Starting 80000 requests...
Completing requests: 100%
Completed requestss: 95%
Client threads complete ... unloading web application
Digest validation failed for stderr.log, expecting 0x9e3bf6dd352112a5d95adcaa8069a8016cba2160 found 0x05cb2bde3a955d45dc057403f0c3419a6e535db3
===== DaCapo 23.9-RC3-chopin tomcat FAILED warmup =====
===== DaCapo simple tail latency: 50% 11677 usec, 90% 41650 usec, 99% 273387 usec, 99.9% 2252914 usec, 99.99% 2312888 usec, max 2320616 usec, measured over 80000 events =====
===== DaCapo metered tail latency: 50% 3638527 usec, 90% 6167409 usec, 99% 6598047 usec, 99.9% 6680881 usec, 99.99% 6702856 usec, max 6714887 usec, measured over 80000 events =====
===== DaCapo 23.9-RC3-chopin tomcat starting warmup 2 =====
Loading web application
Starting 80000 requests...
Completing requests: 100%
Completed requestss: 95%
Client threads complete ... unloading web application
Digest validation failed for stderr.log, expecting 0x9e3bf6dd352112a5d95adcaa8069a8016cba2160 found 0xb21379d567c50f0f7c9bb2de387ac2f88cfa663a
===== DaCapo 23.9-RC3-chopin tomcat FAILED warmup =====
===== DaCapo simple tail latency: 50% 9752 usec, 90% 30288 usec, 99% 103893 usec, 99.9% 228910 usec, 99.99% 281012 usec, max 310636 usec, measured over 80000 events =====
===== DaCapo metered tail latency: 50% 30617 usec, 90% 413548 usec, 99% 516381 usec, 99.9% 598007 usec, 99.99% 647202 usec, max 696496 usec, measured over 80000 events =====
===== DaCapo 23.9-RC3-chopin tomcat starting =====
Loading web application
Starting 80000 requests...
Completing requests: 100%
Completed requests
Client threads complete ... unloading web application
Digest validation failed for stderr.log, expecting 0x9e3bf6dd352112a5d95adcaa8069a8016cba2160 found 0xec92af78c9f4b14b2eb48ee63758062c12d148c5
===== DaCapo 23.9-RC3-chopin tomcat FAILED =====
===== DaCapo simple tail latency: 50% 9977 usec, 90% 36023 usec, 99% 113435 usec, 99.9% 217206 usec, 99.99% 277376 usec, max 351223 usec, measured over 80000 events =====
===== DaCapo metered tail latency: 50% 15496 usec, 90% 66028 usec, 99% 161969 usec, 99.9% 253358 usec, 99.99% 327533 usec, max 361036 usec, measured over 80000 events =====
Server stopped ... iteration complete
Validation FAILED for tomcat default
The small size works just fine:
$ ../JAVA/jdk-17.0.8.1+1/bin/java -jar dacapo-23.9-RC3-chopin.jar -v -n 3 tomcat -s small
The derived number of threads (160) is outside the range [1,4]; rescaling to match thread limit.
Class name: org.dacapo.harness.Tomcat
Configurations:
short Tomcat servlet container
long Runs the tomcat sample web applications
author Apache Software Foundation
license Apache License, Version 2.0
copyright Copyright (c) Apache Software Foundation
url http://tomcat.apache.org
version tomcat 10.1.11
sizes default large small
Using scaled threading model. 160 processors detected, 4 threads used to drive the workload, in a possible range of [1,4]
Version: tomcat 10.1.11 (use -p to print nominal benchmark stats)
Server thread created
===== DaCapo 23.9-RC3-chopin tomcat starting warmup 1 =====
Loading web application
Starting 160 requests...
Completed requests
Client threads complete ... unloading web application
===== DaCapo 23.9-RC3-chopin tomcat completed warmup 1 in 6255 msec =====
===== DaCapo simple tail latency: 50% 107755 usec, 90% 216129 usec, 99% 1673361 usec, 99.9% 1673453 usec, 99.99% 1673453 usec, max 1673453 usec, measured over 160 events =====
===== DaCapo metered tail latency: 50% 1737097 usec, 90% 2228633 usec, 99% 2359272 usec, 99.9% 2381082 usec, 99.99% 2381082 usec, max 2381082 usec, measured over 160 events =====
===== DaCapo 23.9-RC3-chopin tomcat starting warmup 2 =====
Loading web application
Starting 160 requests...
Completed requests
Client threads complete ... unloading web application
===== DaCapo 23.9-RC3-chopin tomcat completed warmup 2 in 891 msec =====
===== DaCapo simple tail latency: 50% 12384 usec, 90% 21163 usec, 99% 69394 usec, 99.9% 72357 usec, 99.99% 72357 usec, max 72357 usec, measured over 160 events =====
===== DaCapo metered tail latency: 50% 39660 usec, 90% 88973 usec, 99% 121264 usec, 99.9% 127067 usec, 99.99% 127067 usec, max 127067 usec, measured over 160 events =====
===== DaCapo 23.9-RC3-chopin tomcat starting =====
Loading web application
Starting 160 requests...
Completed requests
Client threads complete ... unloading web application
===== DaCapo 23.9-RC3-chopin tomcat PASSED in 826 msec =====
===== DaCapo simple tail latency: 50% 10526 usec, 90% 20619 usec, 99% 48546 usec, 99.9% 49080 usec, 99.99% 49080 usec, max 49080 usec, measured over 160 events =====
===== DaCapo metered tail latency: 50% 48143 usec, 90% 100765 usec, 99% 122895 usec, 99.9% 127535 usec, 99.99% 127535 usec, max 127535 usec, measured over 160 events =====
Server stopped ... iteration complete
Large size is also ok:
$ ../JAVA/jdk-17.0.8.1+1/bin/java -jar dacapo-23.9-RC3-chopin.jar -v -n 3 tomcat -s large
Class name: org.dacapo.harness.Tomcat
Configurations:
short Tomcat servlet container
long Runs the tomcat sample web applications
author Apache Software Foundation
license Apache License, Version 2.0
copyright Copyright (c) Apache Software Foundation
url http://tomcat.apache.org
version tomcat 10.1.11
sizes default large small
Using scaled threading model. 160 processors detected, 160 threads used to drive the workload, in a possible range of [1,4096]
Version: tomcat 10.1.11 (use -p to print nominal benchmark stats)
Server thread created
===== DaCapo 23.9-RC3-chopin tomcat starting warmup 1 =====
Loading web application
Starting 800000 requests...
Completing requests: 100%
Completed requests
Client threads complete ... unloading web application
===== DaCapo 23.9-RC3-chopin tomcat completed warmup 1 in 86607 msec =====
===== DaCapo simple tail latency: 50% 12779 usec, 90% 30367 usec, 99% 122404 usec, 99.9% 338153 usec, 99.99% 2104715 usec, max 2216072 usec, measured over 800000 events =====
===== DaCapo metered tail latency: 50% 3657862 usec, 90% 6057514 usec, 99% 7536661 usec, 99.9% 7612415 usec, 99.99% 7697998 usec, max 7831637 usec, measured over 800000 events =====
===== DaCapo 23.9-RC3-chopin tomcat starting warmup 2 =====
Loading web application
Starting 800000 requests...
Completing requests: 100%
Completed requests
Client threads complete ... unloading web application
===== DaCapo 23.9-RC3-chopin tomcat completed warmup 2 in 82888 msec =====
===== DaCapo simple tail latency: 50% 13913 usec, 90% 31567 usec, 99% 111083 usec, 99.9% 186958 usec, 99.99% 239837 usec, max 334088 usec, measured over 800000 events =====
===== DaCapo metered tail latency: 50% 1301760 usec, 90% 1446867 usec, 99% 1500901 usec, 99.9% 1582263 usec, 99.99% 1637113 usec, max 1720613 usec, measured over 800000 events =====
===== DaCapo 23.9-RC3-chopin tomcat starting =====
Loading web application
Starting 800000 requests...
Completing requests: 100%
Completed requests
Client threads complete ... unloading web application
===== DaCapo 23.9-RC3-chopin tomcat PASSED in 74462 msec =====
===== DaCapo simple tail latency: 50% 11485 usec, 90% 31641 usec, 99% 96594 usec, 99.9% 187849 usec, 99.99% 244871 usec, max 362868 usec, measured over 800000 events =====
===== DaCapo metered tail latency: 50% 1596831 usec, 90% 1994489 usec, 99% 2070355 usec, 99.9% 2126507 usec, 99.99% 2224566 usec, max 2316226 usec, measured over 800000 events =====
Server stopped ... iteration complete
@heshpdx, could you please attach the stderr.log?
It will be in scratch/stderr.log. You will need to use the -preserve command line argument to preserve the scratch directory (its contents are deleted at the end of execution, by default).
after the failing run:
$ cat scratch/stderr.log
Starting 80000 requests...
Completing requests: 100%
Completed requests
$ cat scratch/stdout.log
Loading web application
Client threads complete ... unloading web application
Attaching the whole scratch directory in case that is useful.
tomcat-scratch.tar.gz
I see the issue.
Completing requests: 100%
Completed requestss: 95%
It seems there's a race condition there. Presumably exposed by your testing at high core counts.
More data points on aarch64.
PASSES:
- Huawei Taishan, openjdk-11, 96 threads
- Huawei Taishan, msjdk-17, 96 threads
- Ampere Altra, openjdk-11, 160 threads
- Ampere Altra, msjdk-17, 111 threads (and anything lower)
FAILS:
- Ampere Altra, msjdk-17, 112 threads (and anything higher, although between 112 and 118 is flaky and sometimes passes, indicating that it is a non-deterministic race condition as you suggested).
Fixed in 0f83db5