Crash on generating a vertical profile
p4block opened this issue · comments
I'm running the latest Docker image and viewing some grid data using godiva2. This used to work ~2 releases ago and I'm unsure what happened.
This picture explains what I'm doing and what the user sees.
This can be found in the logs:
2020-02-28T23:43:03.870 +0000 [ 225909][ 239] INFO - threddsServlet - Remote host: 82.167.5.3 - Request: "GET /thredds/wms/L2/Salinidad.nc?REQUEST=GetVerticalProfile&LAYER=Salinidad&
CRS=CRS:84&TIME=2020-02-11T00:00:00.000Z&POINT=-0.8120637892857303%2037.74341714166666&FORMAT=image/png HTTP/1.1"
2020-02-28T23:43:03.874 +0000 [ 225913][ 239] ERROR - thredds.server.wms.ThreddsWmsController - dispatchWmsRequest(): Error:
java.lang.NoClassDefFoundError: Could not initialize class sun.font.SunFontManager
at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:264) ~[?:1.8.0_242]
at sun.java2d.SunGraphics2D.getFontMetrics(SunGraphics2D.java:855) ~[?:1.8.0_242]
at org.jfree.text.G2TextMeasurer.getStringWidth(G2TextMeasurer.java:79) ~[jcommon-1.0.23.jar:?]
at org.jfree.text.TextUtilities.nextLineBreak(TextUtilities.java:306) ~[jcommon-1.0.23.jar:?]
at org.jfree.text.TextUtilities.createTextBlock(TextUtilities.java:247) ~[jcommon-1.0.23.jar:?]
at org.jfree.chart.title.TextTitle.arrangeRR(TextTitle.java:628) ~[jfreechart-1.0.19.jar:?]
at org.jfree.chart.title.TextTitle.arrange(TextTitle.java:496) ~[jfreechart-1.0.19.jar:?]
at org.jfree.chart.JFreeChart.drawTitle(JFreeChart.java:1311) ~[jfreechart-1.0.19.jar:?]
at org.jfree.chart.JFreeChart.draw(JFreeChart.java:1203) ~[jfreechart-1.0.19.jar:?]
at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1399) ~[jfreechart-1.0.19.jar:?]
at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1379) ~[jfreechart-1.0.19.jar:?]
at org.jfree.chart.ChartUtilities.writeChartAsPNG(ChartUtilities.java:184) ~[jfreechart-1.0.19.jar:?]
at org.jfree.chart.ChartUtilities.writeChartAsPNG(ChartUtilities.java:138) ~[jfreechart-1.0.19.jar:?]
at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.getVerticalProfile(AbstractWmsController.java:1030) ~[ncwms-1.2.tds.4.6.14.jar:1.2.tds.4.6.14]
at thredds.server.wms.ThreddsWmsController.dispatchWmsRequest(ThreddsWmsController.java:203) [classes/:4.6.14]
at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.handleRequestInternal(AbstractWmsController.java:207) [ncwms-1.2.tds.4.6.14.jar:1.2.tds.4.6.14]
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) [servlet-api.jar:?]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [servlet-api.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:118) [classes/:4.6.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:364) [catalina.jar:8.5.51]
at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:170) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at thredds.servlet.filter.RequestCORSFilter.doFilterInternal(RequestCORSFilter.java:49) [classes/:4.6.14]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:94) [classes/:4.6.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at thredds.server.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:81) [classes/:4.6.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [log4j-web-2.8.2.jar:2.8.2]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.51]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.51]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.51]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.51]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) [catalina.jar:8.5.51]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [catalina.jar:8.5.51]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.51]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:688) [catalina.jar:8.5.51]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.51]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) [catalina.jar:8.5.51]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [catalina.jar:8.5.51]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609) [tomcat-coyote.jar:8.5.51]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote.jar:8.5.51]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) [tomcat-coyote.jar:8.5.51]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623) [tomcat-coyote.jar:8.5.51]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.51]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
What could be happening?
Do you know which version of Java you are using inside the image?
$ java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
@p4block it should theoretically be easy to pull in an earlier version of the thredds-dockercontainer in your setup (e.g., v4.6.10) instead of latest or whatever you have now. Can you confirm the version for which it was still working?
I see the same behavior on thredds-jetstream. The error message in the logs indicate the base java image is missing something, and I'm seeing similar reports elsewhere when using openjdk base images.
Potentially related: javamelody/javamelody#819
I've seen apt-get install fontconfig referenced in many places as being a solution to the problem (also mentioned in the javamelody issue).
@lesserwhirls can you confirm this issue is now fixed here: https://tds.scigw.unidata.ucar.edu/thredds/catalog.html ?
Though I am seeing something different now when trying to do a vertical profile:
Could this be a configuration problem?
Thanks for the quick response!
@julienchastang The issue on Chrome shows like that, Firefox shows the error like on the first screenshot. I tried to install fontconfig on the container and it was already installed, so the pull request probably does nothing.
I downgraded to 4.6.10 and the vertical profile worked (!)
Also tried 4.6.13 and it also worked.
Specifying 4.6.14 or latest gets the issue back.
Hmm. Was there a tomcat or Java change in the docker images at the same time the image bumped to 4.6.14? I am able to generate profiles on other site running 4.6.14, for example, on the Pacific Islands Ocean Observing System TDS:
There are a ton of 4.6.14 images uploaded to docker hub, the latest of them is broken for sure but I haven't tried any of the others. The issue is probably unrelated to TDS itself and is caused by some innocent Dockerfile change.
I wouldn't really call it broken but limited in this specific way. Thanks for finding the problem. Yes, versions of Tomcat (and therefore Java underneath) "evolve" for each TDS Docker image. Remember this container is built on top of a stock Tomcat image so whatever changes are going on there get passed down to the TDS docker image. The Docker image for TDS 4.6.14, for example, has been updated a few times to accommodate newer versions of Tomcat. This is mandated by security concerns largely. Running outdated versions of Tomcat will get us flagged.
This scheme, BTW, is the same for, say, the Tomcat image we reference. I always reference the same Tomcat image (e.g., tomcat:8.5-jdk8-openjdk) even though it too is "evolving". In the end, it makes it difficult to do git bisect type operations to find the source of problems.
I'll keep digging into this when I have time to do so.
Any chance we could try tomcat's 8.5-jdk8-adoptopenjdk-hotspot image? I've read conversations where people say the adoptopenjdk image has better support across various platforms (Ubuntu was specifically mentioned).
I ran into this issue. But from what I can tell, we are not using the slim jdk. At present, I believe our current set up should work. If it doesn't, it means backwards compatibly was not respected in base images. That's the root problem, IMO.
Separately, I wonder if we are chasing a red herring. When trying to do the vertical profile, I saw this error:
java.lang.UnsatisfiedLinkError: /usr/local/openjdk-8/jre/lib/amd64/libfontmanager.so: /usr/local/lib/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/lib/x86_64-linux-gnu/libpng16.so.16)
at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_242]
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1946) ~[?:1.8.0_242]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1849) ~[?:1.8.0_242]
at java.lang.Runtime.loadLibrary0(Runtime.java:871) ~[?:1.8.0_242]
at java.lang.System.loadLibrary(System.java:1124) ~[?:1.8.0_242]
Is zlib the source of the problem here instead? zlib, currently v1.2.8, is actually installed in this container for netcdf purposes. Let me test out that theory.
Interesting. We don't need to have v1.2.8 specifically for netCDF-C - I'm sure we can rely on what is available transitively.
Nice catch, BTW
@lesserwhirls @p4block I believe #236 may work. Can you confirm at https://tds.scigw.unidata.ucar.edu/thredds/catalog.html
@WardF @DennisHeimbigner somewhat out of left-field, is there any problem with me using zlib v1.2.9 for the netcdf build? I believe v1.2.8 is the officially sanctioned version for netcdf. The netcdf build process did not seem to complain, but thought I would check. I am encountering a zlib version conflict due to an independent fontconfig zlib dependency herein.
Seems to be working from my end. Thanks @julienchastang!
AFAIK, that zlib should be ok.
1.2.8 is the minimum version supported, but 1.2.9 should work just fine; if it's truly an error using v1.2.9, I would appreciate somebody pinging me to open an issue, or opening one, over at https://github.com/Unidata/netcdf-c.
1.2.8 is the minimum version supported, but 1.2.9 should work just fine; if it's truly an error using v1.2.9, I would appreciate somebody pinging me to open an issue, or opening one, over at https://github.com/Unidata/netcdf-c.
In this case, it's not an error related to netCDF-C. Here we have one docker image (openjdk) looking for zlib 1.2.9, and the TDS docker image (which builds on top of an image that builds on top of openjdk) installing 1.2.8 to use with netCDF-C (which made openjdk mad because 1.2.9 was no longer there). Julien was just checking to make sure 1.2.9 was ok to use with netCDF-C, which it is, so we should be good.
That did it, it's working again in our setup. Thanks a lot, incredible work.