eclipse/steady-rest-lib-utils crash with OutOfMemoryError

Question

eclipse/steady-rest-lib-utils crash with OutOfMemoryError

zhaolida98 opened this issue 3 years ago · comments

Hi, I deployed Steady on my computer with the docker images according to this guide [1]. Everything works fine within the first 30 mins to 1 hour. After that, when I check http://localhost:8033/haproxy?stats, I can see rest-lib-utils-nodes is down.

however, the docker container is still running

When I check the log inside, I found it was crashed with OutOfMemoryError.

java.lang.OutOfMemoryError: Java heap space
	at org.springframework.boot.loader.data.RandomAccessDataFile.read(RandomAccessDataFile.java:101) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.CentralDirectoryParser.parseEntries(CentralDirectoryParser.java:64) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:57) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:139) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:123) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:109) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:100) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.Handler.getRootJarFile(Handler.java:385) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.Handler.getRootJarFileFromUrl(Handler.java:373) ~[rest-lib-utils.jar:3.2.0]
	at org.springframework.boot.loader.jar.Handler.openConnection(Handler.java:92) ~[rest-lib-utils.jar:3.2.0]
	at java.base/java.net.URL.openConnection(Unknown Source) ~[na:na]
	at javassist.JarClassPath.openClassfile(ClassPoolTail.java:163) ~[javassist-3.28.0-GA.jar!/:na]
	at javassist.ClassPoolTail.openClassfile(ClassPoolTail.java:340) ~[javassist-3.28.0-GA.jar!/:na]
	at javassist.ClassPool.openClassfile(ClassPool.java:621) ~[javassist-3.28.0-GA.jar!/:na]
	at javassist.CtClassType.getClassFile3(CtClassType.java:217) ~[javassist-3.28.0-GA.jar!/:na]
	at javassist.CtClassType.getClassFile2(CtClassType.java:178) ~[javassist-3.28.0-GA.jar!/:na]
	at javassist.CtClassType.getModifiers(CtClassType.java:458) ~[javassist-3.28.0-GA.jar!/:na]
	at javassist.CtClassType.isInterface(CtClassType.java:443) ~[javassist-3.28.0-GA.jar!/:na]
	at org.eclipse.steady.java.JarAnalyzer.getConstructIds(JarAnalyzer.java:501) ~[lang-java-3.2.0.jar!/:na]
	at org.eclipse.steady.java.JarAnalyzer.getSharedConstructs(JarAnalyzer.java:702) ~[lang-java-3.2.0.jar!/:na]
	at org.eclipse.steady.cia.rest.ArtifactController.intersect(ArtifactController.java:488) ~[classes!/:3.2.0]
	at jdk.internal.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.8.jar!/:5.3.8]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.8.jar!/:5.3.8]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.8.jar!/:5.3.8]

full log is here:
steady-rest-lib-utils.log

Surprisingly, I can still process a scan even when steady-rest-lib-util is down.

Then I tried to pull the newest steady git repo and up again. This time I did nothing but waiting. The steady-rest-lib-utils crashed for the same reason within about 1.5 hours.

Could you please help me check why is this happening? Will this affect the functionality of the scan results(especially cve update and output)?

Regards.

[1] https://eclipse.github.io/steady/admin/tutorials/docker/#setup

Serena Ponta · Answer 1 · Tue Jan 18 2022 01:16:16 GMT+0800 (China Standard Time)

Hello @zhaolida98

thank you for trying Steady and sorry for taking so long to come back to you.

The steady-rest-lib-util service offers utility functionalities for libraries and it is not directly used during the scan of applications (for all goals but checkcode). This is why you can still scan the application even when the steady-rest-lib-util service is down or not working. That service is mainly used to establish whether libraries containing constructs modified to fix a vulnerability contain the vulnerable or fixed version of the code. Thus, when it's not working, you would get "orange hourglasses" in the results highlighting the facts that it is still unknown whether the dependencies contain the vulnerable or the fixed version of the code (See bottom of [2]).

To get more insights on what is going on, could you share the logs of steady-kb-importer (that imports and updates the vulnerabilities) and steady-patch-lib-analyzer (that invokes steady-rest-lib-util for the task described above)?

Thank you!

[2] https://eclipse.github.io/steady/vuln_db/

NRYet · Answer 2 · Tue Jan 18 2022 09:36:41 GMT+0800 (China Standard Time)

OK I see, so when steady-rest-lib-util is down, the scan will continue but the ability to detect vulnerability (lets just say CVEs ) are largely impaired, right? Anyway,

here is the log from `steady-patch-lib-analyzer`

2021-12-24 05:40:21,475 [pool-162-thread-4] [INFO ] tcheval.LibraryAnalyzerThread2 - ++++++++Thread 10 for library id [org.apache.hbase|hbase-shaded-client|3.0.0-alpha-1] finished+++++++++
2021-12-24 05:40:21,475 [pool-162-thread-4] [INFO ] tcheval.LibraryAnalyzerThread2 - Analysis of lib [[org.apache.hbase|hbase-shaded-client|2.4.7]] with tid [15]
2021-12-24 05:40:21,475 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP GET [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/1.7.1?skipResponseBody=true&classifier=sources&packaging=jar]
2021-12-24 05:40:22,690 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP GET completed with response code [200] in [01.214 ms] (proxy=false)
2021-12-24 05:40:22,691 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP POST [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/1.7.1/jar/constructIds/intersect?lang=JAVA, size=0.48 KB]
2021-12-24 05:41:08,212 [pool-162-thread-2] [INFO ] kend.requests.BasicHttpRequest - HTTP POST completed with response code [200] in [45.521 ms] (proxy=false)
2021-12-24 05:41:08,213 [pool-162-thread-2] [INFO ] tcheval.LibraryAnalyzerThread2 - Qname [org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(InputStream,OutputStream)] not in sources of [[org.apache.hbase|hbase-shaded-client|1.7.1]]
2021-12-24 05:41:08,213 [pool-162-thread-2] [INFO ] tcheval.LibraryAnalyzerThread2 - Qname [org.apache.hadoop.hbase.ipc.SecureClient(Class,Configuration,SocketFactory)] not in sources of [[org.apache.hbase|hbase-shaded-client|1.7.1]]
2021-12-24 05:41:08,213 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7?skipResponseBody=true&packaging=jar]
2021-12-24 05:41:08,213 [pool-162-thread-2] [INFO ] tcheval.LibraryAnalyzerThread2 - cids contains [[null:JAVA|METH|org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(InputStream,OutputStream)]], change type [MOD]
2021-12-24 05:41:09,476 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET completed with response code [200] in [01.263 ms] (proxy=false)
2021-12-24 05:41:09,476 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7?skipResponseBody=true&classifier=sources&packaging=jar]
2021-12-24 05:41:10,598 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP GET completed with response code [200] in [01.121 ms] (proxy=false)
2021-12-24 05:41:10,599 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA, size=0.48 KB]
2021-12-24 06:11:10,208 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST completed with response code [500] in [000:29:59] (proxy=false)
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [X-Vulas-Version] = 3.2.0
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [X-Vulas-Component] = client
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [Content-Type] = application/json; charset=utf-8
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [Content-Language] = en-US
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Vary] = Origin
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Vary] = Access-Control-Request-Method
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Vary] = Access-Control-Request-Headers
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Content-Type] = application/json
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Transfer-Encoding] = chunked
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Date] = Fri, 24 Dec 2021 06:11:10 GMT
2021-12-24 06:11:10,208 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Connection] = close
2021-12-24 06:11:10,209 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-body: [{"timestamp":1640326270207,"status":500,"error":"Internal Server Error","path":"/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect"}]
2021-12-24 06:11:10,209 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Exception message: [Got error [500] when calling [POST] on [http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA]]
2021-12-24 06:11:10,209 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     HTTP Request body: [[{"lang":"JAVA","type":"METH","qname":"org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(InputStream,OutputStream)","relates":null,"attributes":null},{"lang":"JAVA","type":"CONS","qname":"org.apache.hadoop.hbase.ipc.SecureClient(Class,Configuration,SocketFactory)","relates":null,"attributes":null},{"lang":"JAVA","type":"METH","qname":"org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(InputStream,OutputStream)","relates":null,"attributes":null}]]
2021-12-24 06:11:10,209 [pool-162-thread-4] [WARN ] tcheval.LibraryAnalyzerThread2 - The intersection returned null (thus something went wrong in cia); the Jar for library Id [[org.apache.hbase|hbase-shaded-client|2.4.7]] will not be included in the csv for MOD constructs
2021-12-24 06:11:10,209 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST [uri=http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA, size=0.30 KB]
2021-12-24 06:13:40,560 [pool-162-thread-4] [INFO ] kend.requests.BasicHttpRequest - HTTP POST completed with response code [500] in [000:02:30] (proxy=false)
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [X-Vulas-Version] = 3.2.0
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [X-Vulas-Component] = client
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [Content-Type] = application/json; charset=utf-8
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Request-header [Content-Language] = en-US
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Vary] = Origin
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Vary] = Access-Control-Request-Method
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Vary] = Access-Control-Request-Headers
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Content-Type] = application/json
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Transfer-Encoding] = chunked
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Date] = Fri, 24 Dec 2021 06:13:40 GMT
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-header [Connection] = close
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Response-body: [{"timestamp":1640326420559,"status":500,"error":"Internal Server Error","path":"/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect"}]
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     Exception message: [Got error [500] when calling [POST] on [http://cia:8092/cia/artifacts/org.apache.hbase/hbase-shaded-client/2.4.7/jar/constructIds/intersect?lang=JAVA]]
2021-12-24 06:13:40,560 [pool-162-thread-4] [ERROR] kend.requests.BasicHttpRequest -     HTTP Request body: [[{"lang":"JAVA","type":"CONS","qname":"org.apache.hadoop.hbase.security.HBaseSaslRpcClient(AuthMethod,Token,String)","relates":null,"attributes":null},{"lang":"JAVA","type":"CONS","qname":"org.apache.hadoop.hbase.security.HBaseSaslRpcClient(AuthMethod,Token,String,boolean)","relates":null,"attributes":null}]]
2021-12-24 06:13:40,560 [pool-162-thread-4] [WARN ] tcheval.LibraryAnalyzerThread2 - The intersection returned null (thus something went wrong in cia); the artifact for library Id [[org.apache.hbase|hbase-shaded-client|2.4.7]] will not be included in the csv for ADD/DEL constructs

The real problem seems to locate in steady-rest-lib-util. It shows an "out of memory" error. Before this error, there are plenty of First parameter in argument XXX does not match the to be skipped parameter [XX]. Don't know what happened, I guess there is some memory leakage?

Here is the log from `steady-rest-lib-util`

``` 2021-12-24 05:57:31.396 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hbase.thirdparty.com.google.common.collect.Tables$1()] 2021-12-24 05:57:34.550 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hbase.thirdparty.com.google.common.io.Files$2()] 2021-12-24 05:57:34.559 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hbase.thirdparty.com.google.common.io.Files$1()] 2021-12-24 05:57:50.204 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$3(CommonPattern)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:57:50.205 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$4(int)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:57:50.210 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$1(CharMatcher)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:57:50.211 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Splitter$2(String)] does not match the to be skipped parameter [Splitter] 2021-12-24 05:58:31.427 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$4(long,int,OfLong,LongFunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:31.428 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$5(long,int,OfDouble,DoubleFunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:34.652 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$2(long,int,Iterator,FunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:34.652 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$3(long,int,OfInt,IntFunctionWithIndex)] does not match the to be skipped parameter [Streams] 2021-12-24 05:58:34.653 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.collect.Streams$1(long,int,Iterator,Iterator,BiFunction)] does not match the to be skipped parameter [Streams] 2021-12-24 06:00:06.078 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hadoop.hbase.shaded.org.apache.curator.framework.schema.SchemaSet$2(List,boolean)] does not match the to be skipped parameter [SchemaSet] 2021-12-24 06:02:14.705 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1()] 2021-12-24 06:02:58.680 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$4(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:02:58.681 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$3(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:02:58.682 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$2(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:02:58.682 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.base.Predicates$ObjectPredicate$1(String,int)] does not match the to be skipped parameter [ObjectPredicate] 2021-12-24 06:04:58.204 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : No parameter to skip in argument [org.apache.hadoop.hbase.shaded.org.apache.http.client.entity.GzipDecompressingEntity$1()] 2021-12-24 06:05:52.156 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.graph.Traverser$1(SuccessorsFunction,SuccessorsFunction)] does not match the to be skipped parameter [Traverser] 2021-12-24 06:05:52.157 WARN 8 --- [io-8092-exec-39] org.eclipse.steady.java.JavaId : First parameter in argument [org.apache.hbase.thirdparty.com.google.common.graph.Traverser$2(SuccessorsFunction,SuccessorsFunction)] does not match the to be skipped parameter [Traverser] 2021-12-24 06:11:10.206 ERROR 8 --- [io-8092-exec-39] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/cia] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause

java.lang.OutOfMemoryError: Java heap space
at org.springframework.boot.loader.data.RandomAccessDataFile.read(RandomAccessDataFile.java:101) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.CentralDirectoryParser.parseEntries(CentralDirectoryParser.java:64) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:57) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.(JarFile.java:139) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.(JarFile.java:123) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.(JarFile.java:109) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.JarFile.(JarFile.java:100) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.Handler.getRootJarFile(Handler.java:385) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.Handler.getRootJarFileFromUrl(Handler.java:373) ~[rest-lib-utils.jar:3.2.0]
at org.springframework.boot.loader.jar.Handler.openConnection(Handler.java:92) ~[rest-lib-utils.jar:3.2.0]
at java.base/java.net.URL.openConnection(Unknown Source) ~[na:na]
at javassist.JarClassPath.openClassfile(ClassPoolTail.java:163) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.ClassPoolTail.openClassfile(ClassPoolTail.java:340) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.ClassPool.openClassfile(ClassPool.java:621) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.getClassFile3(CtClassType.java:217) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.getClassFile2(CtClassType.java:178) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.getModifiers(CtClassType.java:458) ~[javassist-3.28.0-GA.jar!/:na]
at javassist.CtClassType.isInterface(CtClassType.java:443) ~[javassist-3.28.0-GA.jar!/:na]
at org.eclipse.steady.java.JarAnalyzer.getConstructIds(JarAnalyzer.java:501) ~[lang-java-3.2.0.jar!/:na]
at org.eclipse.steady.java.JarAnalyzer.getSharedConstructs(JarAnalyzer.java:702) ~[lang-java-3.2.0.jar!/:na]
at org.eclipse.steady.cia.rest.ArtifactController.intersect(ArtifactController.java:488) ~[classes!/:3.2.0]
at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.8.jar!/:5.3.8]

2021-12-24 06:13:40.555 INFO 8 --- [nio-8092-exec-2] o.s.core.annotation.MergedAnnotation : Failed to introspect annotations on org.springframework.core.annotation.AnnotatedElementUtils$AnnotatedElementForAnnotations@290f8938: java.lang.OutOfMemoryError: Java heap space
2021-12-24 06:13:40.557 ERROR 8 --- [io-8092-exec-26] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/cia] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause

java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-nio-8092-Acceptor" java.lang.OutOfMemoryError: Java heap space
2021-12-24 06:13:40.559 ERROR 8 --- [io-8092-exec-22] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/cia] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause

java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Unknown Source) ~[na:na]
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source) ~[na:na]
at java.base/java.lang.AbstractStringBuilder.append(Unknown Source) ~[na:na]
at java.base/java.lang.StringBuilder.append(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData$LocaleDataStrategy.getCandidateLocales(Unknown Source) ~[na:na]
at java.base/sun.util.resources.Bundles.loadBundleOf(Unknown Source) ~[na:na]
at java.base/sun.util.resources.Bundles.of(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData$1.run(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData$1.run(Unknown Source) ~[na:na]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[na:na]
at java.base/sun.util.resources.LocaleData.getBundle(Unknown Source) ~[na:na]
at java.base/sun.util.resources.LocaleData.getDateFormatData(Unknown Source) ~[na:na]
at java.base/java.text.DateFormatSymbols.initializeData(Unknown Source) ~[na:na]
at java.base/java.text.DateFormatSymbols.(Unknown Source) ~[na:na]
at java.base/sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(Unknown Source) ~[na:na]
at java.base/java.text.DateFormatSymbols.getProviderInstance(Unknown Source) ~[na:na]
at java.base/java.text.DateFormatSymbols.getInstance(Unknown Source) ~[na:na]
at java.base/java.util.Calendar.getDisplayName(Unknown Source) ~[na:na]
at java.base/java.text.SimpleDateFormat.subFormat(Unknown Source) ~[na:na]
at java.base/java.text.SimpleDateFormat.format(Unknown Source) ~[na:na]
at java.base/java.text.SimpleDateFormat.format(Unknown Source) ~[na:na]
at java.base/java.text.DateFormat.format(Unknown Source) ~[na:na]
at org.apache.tomcat.util.http.ConcurrentDateFormat.format(ConcurrentDateFormat.java:52) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHttpDateFormat.java:107) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:923) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:379) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.coyote.Response.action(Response.java:211) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.coyote.Response.sendHeaders(Response.java:438) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:292) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[tomcat-embed-core-9.0.46.jar!/:na]
at java.base/java.io.FilterOutputStream.flush(Unknown Source) ~[na:na]

2021-12-24 06:13:40.564 ERROR 8 --- [io-8092-exec-22] o.a.c.c.C.[Tomcat].[localhost] : Exception Processing ErrorPage[errorCode=0, location=/error]

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273) ~[tomcat-embed-core-9.0.46.jar!/:na]
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[tomcat-embed-core-9.0.46.jar!/:na]
at java.base/java.io.FilterOutputStream.flush(Unknown Source) ~[na:na]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1193) ~[jackson-core-2.12.3.jar!/:2.12.3]
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1008) ~[jackson-databind-2.12.3.jar!/:2.12.3]
....

</details>

Serena Ponta · Answer 3 · Tue Jan 18 2022 21:59:30 GMT+0800 (China Standard Time)

Correct, when it's down the ability to detect CVEs is impacted in terms of false positives (i.e., you may have CVEs reported with 'orange hourglasses' for dependencies already containing the fixed code). On top of it, it may also impact the import of CVEs: rest-lib-utils is used during the import depending on the content of the statement.yaml from https://github.com/SAP/project-kb. This is why I wanted to also have a look at the log from steady-kb-importer to see whether the initial import worked. In fact, the first time you start the composition, the initial import of CVEs starts and that may imply an heavy usage of the steady-rest-lib-utils service. Could you share the log of steady-kb-importer ?
I never experienced an "out of memory" on steady-rest-lib-util running with the default memory options of Docker desktop on a machine with 16G, could you share the result in terms of memory (MEM USAGE / LIMIT) of docker stats?

NRYet · Answer 4 · Tue Jan 18 2022 23:26:04 GMT+0800 (China Standard Time)

Sorry, I miss the log of steady-kb-importer, it is surprisingly short here it is:

  You have the latest version.
  Fri Dec 24 05:07:59 UTC 2021  Kaybee Import already Running
  no crontab for root
  cron job created.

and my docker stats is:

CONTAINER ID   NAME                        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
759f973115ad   steady-patch-lib-analyzer   0.04%     421.4MiB / 15.66GiB   2.63%     22.6MB / 14.5MB   42.1MB / 20.5kB   27
ce25e92e0d59   steady-kb-importer          0.00%     16.12MiB / 15.66GiB   0.10%     238kB / 1.82kB    55.8MB / 12.3kB   2
3b54a11b6141   steady-haproxy              0.10%     12.42MiB / 15.66GiB   0.08%     14.4GB / 2.22GB   3.08MB / 0B       5
5186e310c9d1   steady-rest-backend         0.27%     1.515GiB / 15.66GiB   9.67%     25.1GB / 4.75GB   65.1MB / 0B       60
e89933a9fcaa   steady-rest-lib-utils       0.04%     4.59GiB / 15.66GiB    29.30%    125MB / 9.56MB    198MB / 13.2MB    40
28a43af0ee0f   steady-frontend-apps        0.10%     104.9MiB / 15.66GiB   0.65%     384MB / 5.57GB    4.7MB / 32.8kB    27
0b95cfddaadf   steady-cache                0.00%     26.65MiB / 15.66GiB   0.17%     1.06GB / 1.21GB   5.61MB / 8.19kB   134
cbb1a24687b3   steady-frontend-bugs        0.10%     99.57MiB / 15.66GiB   0.62%     384MB / 4.84GB    8.04MB / 32.8kB   27
8f789168d712   steady-postgresql           0.00%     242MiB / 15.66GiB     1.51%     1.21GB / 23.8GB   7.8GB / 6.24GB    57

Just want to make sure, learned from previous issues[1], the kb-importer will automatically import the new CVEs every day, right?

[1]#500 (comment)

NRYet · Answer 5 · Thu Jan 27 2022 09:52:57 GMT+0800 (China Standard Time)

@serenaponta Hi, did you figure out why is this problem coming? How to solve it?

Serena Ponta · Answer 6 · Thu Jan 27 2022 19:47:13 GMT+0800 (China Standard Time)

Hi @zhaolida98 ,

your docker stats limits are in line with the ones of the containers we are running and, unfortunately, I was not able to reproduce the Out of Memory issue with rest-lib-utils. We are usually running it on Ubuntu 18.04 and I would recommend using Ubuntu - if possible and if it's not already the case.

From the logs of steady-kb-importer I think the initial import of vulnerabilities from https://github.com/SAP/project-kb didn't complete successfully (even independently from rest-lib-utils). The initial import is time consuming and if it does not complete before the container is stopped, the import state remains "stuck" (there is a running flag whose management needs to be improved).

I would suggest that you delete the folder /docker/kb-importer/data and ensure that .env contains KB_IMPORTER_SKIP_CLONE=True. Upon the first startup, please plan at least 2h for the initial import with such flag (it takes ~2h30 while running "alone" on a machine with 8core and 16Gb of RAM). Once done the log of steady-kb-importer looks as follows
Tue Jan 25 15:35:46 UTC 2022 Kaybee Import Done cron job created.
You can also check how many vulnerabilities are already imported during the import accessing the endpoint http://localhost:8033/backend/bugs (500 vulnerabilities should be there once completed with KB_IMPORTER_SKIP_CLONE=True).

NRYet · Answer 7 · Thu Jan 27 2022 20:04:45 GMT+0800 (China Standard Time)

Hi @serenaponta ,

Thanks for your reply. However, I checked http://localhost:8033/backend/bugs, though the log is not expected, I can see 722 bugs there.

Actually, I am using Ubuntu 1804. In order to get rid of the outOfMemory, now I decide to reinstall the steady using the stable version 3.2.2. I found three docker compose files docker-compose.build.yml docker-compose-new.yml and docker-compose.yml, what is the difference between them? Which one should I use? I used to use docker-compose.yml, is it correct?

Serena Ponta · Answer 8 · Thu Jan 27 2022 20:29:22 GMT+0800 (China Standard Time)

Hi @zhaolida98,

from the number of vulnerabilities you have it looks like the flag KB_IMPORTER_SKIP_CLONE=True in your .env is not present, which is not a bad thing, it just means that the initial import takes far longer. The fact that the kb-importer state is now "stuck" can be seen also by the fact that it didn't import the newly published vulnerabilities as they are 727 as of today. In case you will reuse an existing data folder in the new deployment, please remove the file docker/kb-importer/data/running if present. (The new release will get rid of such problem.)

As for the docker compose files:

docker-compose.build.yml is the one to use when building new docker images starting from the sources (e.g., if you want to build images for version 3.2.3-SNAPSHOT) as described at [1].
docker-compose.yml is the one to start the composition using the docker images available in docker hub.
docker-compose-new.yml is a new compose file used by the script docker/setup-steady.sh that should make it easier to run steady and would allow to use different profiles to run only a subset of containers depending on the use case. As this is not yet documented, I would recommend you to continue using docker-compose.yml.

[1] https://eclipse.github.io/steady/admin/tutorials/build/#building-docker-images-from-source

Serena Ponta · Answer 9 · Mon Sep 19 2022 20:27:26 GMT+0800 (China Standard Time)

Closed due to lack of feedback from issue author