MemberToMetricMappings is not cleaned on app redeploy/shutdown
spyrkob opened this issue · comments
Renaming a metric name (eg @counted(name="mymetric2") to @counted(name="mymetric3")) and redeploying the application results in:
13:28:24,595 ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to /test/rest: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: SRMET00002: No metric of type counter and ID MetricID{name='org.redhat.set.metrics.TestBean.mymetric2', tags=[]} found in registry
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:82)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:346)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:193)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:457)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
This seems to be introduced in 3.0.0 - the MetricRegistries#cleanUp method is no longer called on application shutdown, which results in stale mappings in MemberToMetricMappings.
I was not able to test this with the main branch
Let's not worry too much about main now, it's switching to Micrometer and WildFly won't be consuming that for quite some time (until it bubbles up to the MicroProfile spec) and it will use a different approach for registries (we will need to think it though too, but that's a different story).
removing the Application registry like MetricRegistries#cleanUp used to do it 2.4.x
To be honest I don't really know how exactly this is made to work in WildFly - AFAIK, there is only one application registry per WildFly instance, not per deployment, so dropping the application registry would drop metrics for all deployments?! So is (was) MetricRegistries#cleanUp really being used during undeploy? (On the SmallRye side, I don't recall any change related to this method between 2.4 and 3.0, it could have been a change on the WF side, but I don't have enough insight into the metrics subsystem to understand it ;) )
Anyway, your fix looks good to me. It removes a deployment's mappings along with its metrics without touching stuff from other deployments. If you verified that it fixes the WF problem, I'm OK with merging it
I suppose you'll also want a new 3.0.x release with it, right?
Thanks @jmartisk!
AFAIK, there is only one application registry per WildFly instance, not per deployment, so dropping the application registry would drop metrics for all deployments?!
I just tested this with two apps and that seems to be the case :)
On the SmallRye side, I don't recall any change related to this method between 2.4 and 3.0, it could have been a change on the WF side
AFAIU this is caused by changes in MetricCdiInjectionExtension#registerMetrics - in 2.4 obtaining the Application registry required CDI creating MetricRegistries bean, that was later disposed off during undeploy calling cleanUp method.
I suppose you'll also want a new 3.0.x release with it, right?
Yes I think so. If there is a release scheduled in future it can probably wait for it though.
I merged your fix, thanks.
I'll do a 3.0.3 release today - there's no release schedule, I just release when something important gets in :)