eventLogDoesNotDeadlockUnderConcurrentLoad flakes
schlosna opened this issue · comments
Follow up from #6225
The com.palantir.atlasdb.keyvalue.api.watch.LockWatchEventLogTest#eventLogDoesNotDeadlockUnderConcurrentLoad test is flaking a lot and causing failures on develop, releases (e.g. 0.703.0) and PRs:
As seen in https://output.circle-artifacts.com/output/job/4a04c674-be62-456d-89cc-d98ba6365f3c/artifacts/4/home/circleci/artifacts/junit/atlasdb-impl-shared/test/classes/com.palantir.atlasdb.keyvalue.api.watch.LockWatchEventLogTest.html the eventLogDoesNotDeadlockUnderConcurrentLoad test currently takes ~11 seconds on Circle so was just flaking due to the 10 second executor shutdown timeout prior to #6225 . Locally on M1 Pro this takes ~3 seconds.
MetricName{safeName=executor.queued-duration, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: count: 546; min: 250.1ms; avg: 274.4ms; p50: 274.2ms; p99: 295.5ms; max: 295.9ms
MetricName{safeName=executor.running, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: 70
MetricName{safeName=tasks, safeTags={}}: count: 1; min: 609.5ms; avg: 609.5ms; p50: 609.5ms; p99: 609.5ms; max: 609.5ms
MetricName{safeName=executor.submitted, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: 200,000 total; ops avg: 317,746.22; 1m: 0.00; 5m: 0.00; 15m: 0.00
MetricName{safeName=executor.duration, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: count: 6,480; min: 0.0ms; avg: 3.1ms; p50: 2.4ms; p99: 12.2ms; max: 19.9ms
MetricName{safeName=executor.queued-duration, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: count: 194,122; min: 254.2ms; avg: 5,552.0ms; p50: 5,471.0ms; p99: 11,043.3ms; max: 11,134.4ms
MetricName{safeName=executor.running, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: 0
MetricName{safeName=tasks, safeTags={}}: count: 1; min: 609.5ms; avg: 609.5ms; p50: 609.5ms; p99: 609.5ms; max: 609.5ms
MetricName{safeName=executor.submitted, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: 200,000 total; ops avg: 17,014.78; 1m: 36,801.78; 5m: 39,338.86; 15m: 39,778.39
MetricName{safeName=executor.duration, safeTags={executor=test, javaVersion=11.0.16.1, libraryName=tritium, libraryVersion=0.51.0}}: count: 200,000; min: 0.0ms; avg: 5.3ms; p50: 3.1ms; p99: 32.0ms; max: 47.7ms