More than ~20 DomainApplications for a single domain name fails
hridder opened this issue · comments
Trying to create more than ~20 DomainApplications for a single domain name causes "too many entity groups in a single transaction" exception. This appears to be due to DomainApplicationIndex.loadActiveApplicationsByDomainName() loading every DomainApplication in order to count the non-deleted ones, or some such related business logic. This adds a bunch of Entity Groups to the transaction, blowing the Datastore limit.
This is not currently a high priority for us as we typically don't see many applications for the same name. See also this domain-registry-users group thread.
google.registry.flows.EppController handleEppCommand: Unexpected failure java.lang.IllegalArgumentException: operating on too many entity groups in a single transaction.
at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:54)
at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:129)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.getFutureWithOptionalTimeout(AsyncDatastoreServiceImpl.java:292)
at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.aggregate(AsyncDatastoreServiceImpl.java:264)
at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.get(AsyncDatastoreServiceImpl.java:235)
at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.get(AsyncDatastoreServiceImpl.java:224)
at com.google.appengine.api.datastore.FutureHelper$TxnAwareFuture.get(FutureHelper.java:182)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.googlecode.objectify.impl.ResultAdapter.now(ResultAdapter.java:34)
at com.googlecode.objectify.impl.Round$2.now(Round.java:135)
at com.googlecode.objectify.impl.Round$2.now(Round.java:132)
at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:172)
at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:164)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.impl.Round$1.nowUncached(Round.java:73)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.LoadResult.now(LoadResult.java:25)
at google.registry.model.ofy.CommitLogBucket.loadBucket(CommitLogBucket.java:128)
at google.registry.model.ofy.CommitLoggedWork.saveCommitLog(CommitLoggedWork.java:134)
at google.registry.model.ofy.CommitLoggedWork.vrun(CommitLoggedWork.java:121)
at com.googlecode.objectify.VoidWork.run(VoidWork.java:14)
at com.googlecode.objectify.VoidWork.run(VoidWork.java:11)
at com.googlecode.objectify.impl.TransactorNo.transactOnce(TransactorNo.java:118)
at com.googlecode.objectify.impl.TransactorNo.transactNew(TransactorNo.java:95)
at com.googlecode.objectify.impl.ObjectifyImpl.transactNew(ObjectifyImpl.java:193)
at com.googlecode.objectify.impl.ObjectifyImpl.transactNew(ObjectifyImpl.java:185)
at google.registry.model.ofy.Ofy.transactCommitLoggedWork(Ofy.java:225)
at google.registry.model.ofy.Ofy.transactNew(Ofy.java:210)
at google.registry.model.ofy.Ofy.transact(Ofy.java:203)
at google.registry.flows.FlowRunner.run(FlowRunner.java:101)
at google.registry.flows.EppController.handleEppCommand(EppController.java:82)
at google.registry.flows.EppServletUtils.handleEppCommandAndWriteResponse(EppServletUtils.java:49)
at google.registry.flows.EppTlsServlet.doPost(EppTlsServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at google.registry.flows.EppOauthFilter.doFilter(EppOauthFilter.java:62)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at google.registry.model.ofy.OfyFilter.doFilter(OfyFilter.java:33)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.googlecode.objectify.cache.AsyncCacheFilter.doFilter(AsyncCacheFilter.java:59)
at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:49)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:50)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:260)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:78)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:148)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:469)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:439)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:446)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:256)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:310)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:302)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:443)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:235)
at java.lang.Thread.run(Thread.java:745)
For Google contributors, we have an internal bug tracking this at b/29998446.
Is this issue still a thing? If so, is there possiblity I can contribute?
I'm sorry I can't help much as I gave up working on Nomulus long ago, and it doesn't seem the maintainers bother keeping github issues updated. 😒 In our case the problem only came up in testing where the same domain was created/deleted every test run. We worked around that.
If it's a problem for you or you are otherwise interested I would look at the DomainApplicationIndex.loadActiveApplicationsByDomainName() function (if it still exists) and also review the original discussion thread for details on the cause and possible fix.
I'll let others say whether it's still a thing, or valuable to them. Regards.
Sorry for the neglect on this, we don't really use github for issue tracking internally, but we should certainly be tending to these.
I don't know whether this is still an issue. However, seeing as how we are in the process of migrating Nomulus from Datastore to Cloud SQL it's probably not worth investigating at this time.
More generally, @kamilkub -- yes, we're happy to accept external contributions. Please have a look at CONTRIBUTING.md for details.
Actually, I missed that @nfelt put in an internal bug number. This appears to have been fixed in September, 2017.
It's worth pointing out that we deleted domain applications from our system entirely a few years back, as they are only needed for end-date sunrise, and we haven't launched an end-date sunrise TLD since 2014 with no plans to do so going forward. The same is true of other registry operators -- most tend to prefer start-date sunrise for its lower operational overhead.