ClaimsListShardTest Occasional test failure

[Wolfgang Meyers]

Our CI system sometimes has the build fail due to com.google.domain.registry.model.tmch.ClaimsListShardTest.test_savesAndGets_withSharding - This test does not fail every time, but fails intermittently:

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: transaction closed
    at com.google.domain.registry.util.Concurrent.transform(Concurrent.java:92)
    at com.google.domain.registry.util.Concurrent.transform(Concurrent.java:50)
    at com.google.domain.registry.model.tmch.ClaimsListShard.save(ClaimsListShard.java:166)
    at com.google.domain.registry.model.tmch.ClaimsListShardTest.test_savesAndGets_withSharding(ClaimsListShardTest.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at com.google.domain.registry.testing.ExceptionRule$1.evaluate(ExceptionRule.java:58)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:670)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.IllegalArgumentException: transaction closed
    at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:54)
    at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:129)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:96)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:88)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:88)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:88)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:88)
    at com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:102)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:88)
    at com.googlecode.objectify.util.FutureHelper.quietGet(FutureHelper.java:26)
    at com.googlecode.objectify.impl.TransactionImpl.commit(TransactionImpl.java:45)
    at com.googlecode.objectify.impl.TransactorNo.transactOnce(TransactorNo.java:119)
    at com.googlecode.objectify.impl.TransactorNo.transactNew(TransactorNo.java:95)
    at com.googlecode.objectify.impl.ObjectifyImpl.transactNew(ObjectifyImpl.java:193)
    at com.googlecode.objectify.impl.ObjectifyImpl.transactNew(ObjectifyImpl.java:185)
    at com.google.domain.registry.model.ofy.Ofy.transactCommitLoggedWork(Ofy.java:224)
    at com.google.domain.registry.model.ofy.Ofy.transactNew(Ofy.java:209)
    at com.google.domain.registry.model.tmch.ClaimsListShard$2.apply(ClaimsListShard.java:170)
    at com.google.domain.registry.model.tmch.ClaimsListShard$2.apply(ClaimsListShard.java:1)
    at com.google.domain.registry.util.Concurrent$1.call(Concurrent.java:84)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at com.google.appengine.tools.development.RequestThreadFactory$1$1$2.run(RequestThreadFactory.java:110)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.google.appengine.tools.development.RequestThreadFactory$1$1.run(RequestThreadFactory.java:107)

[Justine Tunney]

It's sort of a datastore API bug. This is supposed to be a ConcurrentModificationException exception, which would trigger Objectify to retry. But there are some circumstances under which multiple threads performing XG transactions will cause that error to show up as IAE. The issue hasn't been fixed yet. We also don't really experience this flake in our CI.

How are you invoking the tests in your CI? Are you running bazel test //... or something? If so, can you ignore the pattern that causes this test to run? Like maybe bazel test //... -//javatests/com/google/domain/registry.model.tmch:ClaimsListShardTest or something.