ehcache occurs resilience issue

Question

ehcache occurs resilience issue

jasonwang3 opened this issue a year ago · comments

"org.ehcache.spi.resilience.StoreAccessException: The element with key '64cbf77821dc7973d6b3c8cf' is too large to be stored in this offheap store. at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1038) at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.put(AbstractOffHeapStore.java:250) at org.ehcache.impl.internal.store.tiering.TieredStore.put(TieredStore.java:108) at org.ehcache.core.Ehcache.doPut(Ehcache.java:94) at org.ehcache.core.EhcacheBase.put(EhcacheBase.java:189) at org.ehcache.jsr107.Eh107Cache.put(Eh107Cache.java:175) at org.springframework.cache.jcache.JCacheCache.put(JCacheCache.java:95) at org.springframework.cache.interceptor.AbstractCacheInvoker.doPut(AbstractCacheInvoker.java:87) at org.springframework.cache.interceptor.CacheAspectSupport$CachePutRequest.apply(CacheAspectSupport.java:837) at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:430) at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:345) at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220) at com.sun.proxy.$Proxy68.getBroadcastForConnector(Unknown Source) at com.everbridge.notification.connector.utils.ConnectorMetricsUtils.lambda$logDequeuedDelay$0(ConnectorMetricsUtils.java:156) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.everbridge.notification.connector.utils.ConnectorMetricsUtils.logDequeuedDelay(ConnectorMetricsUtils.java:151) at com.everbridge.notification.connector.ConnectorConsumerActor.getAttempts(ConnectorConsumerActor.java:62) at com.everbridge.notification.connector.ConnectorConsumerActor.lambda$createReceive$0(ConnectorConsumerActor.java:47) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) at akka.actor.ActorCell.invoke(ActorCell.scala:548) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.ehcache.shadow.org.terracotta.offheapstore.exceptions.OversizeMappingException: Storage Engine and Eviction Failed - Everything Pinned (2246 mappings) Storage Engine : org.ehcache.shadow.org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine@5dbb651b at org.ehcache.shadow.org.terracotta.offheapstore.AbstractOffHeapClockCache.storageEngineFailure(AbstractOffHeapClockCache.java:86) at org.ehcache.shadow.org.terracotta.offheapstore.OffHeapHashMap.writeEntry(OffHeapHashMap.java:685) at org.ehcache.shadow.org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1958) at org.ehcache.shadow.org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:520) at org.ehcache.shadow.org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:739) at org.ehcache.impl.internal.store.disk.EhcachePersistentConcurrentOffHeapClockCache.compute(EhcachePersistentConcurrentOffHeapClockCache.java:158) at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1036) ... 37 common frames omitted

jhouserizer · Answer 1 · Thu Aug 24 2023 01:29:55 GMT+0800 (China Standard Time)

Hi @jasonwang3 , can you please provide more information. What does your cache configuration look like? What objects are you storing? Do you have a resilience strategy configured? There's not much we can go forward with given just a stack trace with no context.

jason wang · Answer 2 · Thu Aug 24 2023 10:05:38 GMT+0800 (China Standard Time)

Hi @jhouserizer , ehcache version is 3.9.9
`

<persistence directory="/tmp/${ROLE}"/>

<cache alias="attachements">
    <resources>
        <heap unit="entries">10000</heap>
        <disk unit="GB" persistent="true">4</disk>
    </resources>
</cache>

<cache alias="broadcastForConnector">
    <resources>
        <heap unit="MB">200</heap>
        <disk unit="GB" persistent="true">1</disk>
    </resources>
</cache>

`

resilience strategy is defalut and object is Java normal Object.
In this situation system memroy free is very low about ~170M but JVM not occurs OOM issue.
I'm very confused that not use pinned method but why shown 'Storage Engine and Eviction Failed - Everything Pinned (2246 mappings)'?

This is spring applicationContext
`

<cache:advice id="cacheAdvice" cache-manager="cacheManager">
    <cache:caching cache="attachements">
        <cache:cacheable method="getNotificationAttachment" key="#p0"/>
    </cache:caching>
    <cache:caching cache="broadcastForConnector">
        <cache:cacheable method="getBroadcastForConnector" key="#p0"/>
        <cache:cache-evict method="getBroadcastForConnector" key="#p0" condition="#result.broadcastSettings.isEnableIncidentZone() and #result.endDate.before(new java.util.Date())" before-invocation="false"/>
    </cache:caching>
</cache:advice>`

jhouserizer · Answer 3 · Thu Aug 24 2023 10:37:16 GMT+0800 (China Standard Time)

SFAIK, pinning would kick in if you're using write-behind. I don't see that in your shared config (but there are other ways to enable it) -- are you using write-behind? if so, then my guess is that the writer is not keeping up.

jason wang · Answer 4 · Thu Aug 24 2023 19:13:35 GMT+0800 (China Standard Time)

So when I occurs this situation how should be handle this issue or avoid this issue? Or some listeners can handle like put failure then clear cache store?

jhouserizer · Answer 5 · Thu Aug 24 2023 19:36:58 GMT+0800 (China Standard Time)

Well, can you confirm one way or another that you are / are not using a writer (e..g write-behind)

jason wang · Answer 6 · Thu Aug 24 2023 19:40:40 GMT+0800 (China Standard Time)

I confirmed spring cache default is write-through

jason wang · Answer 7 · Thu Aug 24 2023 20:33:03 GMT+0800 (China Standard Time)

I also found when occurs this issue, ehcache put will become very slow

jhouserizer · Answer 8 · Fri Aug 25 2023 23:06:02 GMT+0800 (China Standard Time)

So this means your writer - writing to the db, isn't not executing fast enough, e.g. it takes longer to write to the db than the interval between your additions/updates to cache entries.

No sure what you can do about that - faster db, less workload, etc.

jason wang · Answer 9 · Sun Aug 27 2023 20:49:15 GMT+0800 (China Standard Time)

What did you mean db? Is disk or memroy?
How can I avoide this issue? Slow down write elements into ehcache throughput? Or can modify some parameters?