Sporadic `ProcessorStateException: Failed to flush state store`

Question

Sporadic `ProcessorStateException: Failed to flush state store`

mewwts opened this issue 7 years ago · comments

I have EmbeddedKafka.start() and .stop() in beforeAll and afterAll, but when I run my suite I see sporadic issues like in this stacktrace:

org.apache.kafka.streams.errors.ProcessorStateException: task [1_0] Failed to flush state store idStore
	at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:262)
	at org.apache.kafka.streams.processor.internals.AbstractTask.flushState(AbstractTask.java:190)
	at org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:282)
	at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:264)
	at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:187)
	at org.apache.kafka.streams.processor.internals.StreamTask.commitImpl(StreamTask.java:259)
	at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:253)
	at org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:815)
	at org.apache.kafka.streams.processor.internals.StreamThread.access$2800(StreamThread.java:73)
	at org.apache.kafka.streams.processor.internals.StreamThread$2.apply(StreamThread.java:797)
	at org.apache.kafka.streams.processor.internals.StreamThread.performOnStreamTasks(StreamThread.java:1448)
	at org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:789)
	at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:778)
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:567)
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
Caused by: java.lang.NullPointerException: null
	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:847)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2975)
	at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:202)
	at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:201)
	at com.schibsted.spt.data.yggdrasil.common.serdes.JsonSerde$$anon$1.readValue(JsonSerde.scala:16)
	at com.schibsted.spt.data.yggdrasil.common.serdes.JsonSerde$JsonDeserializer.deserialize(JsonSerde.scala:35)
	at org.apache.kafka.streams.state.StateSerdes.valueFrom(StateSerdes.java:163)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.putAndMaybeForward(CachingKeyValueStore.java:90)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.access$000(CachingKeyValueStore.java:34)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore$1.apply(CachingKeyValueStore.java:78)
	at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:145)
	at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:103)
	at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:97)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:107)
	at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:260)
	... 14 common frames omitted
2017-09-19 12:38:06,163 ERROR [69d6d46a-3cb4-431e-9342-6305af1f73be-fa6a6ac0-8d20-403b-bd91-8c3497ada04f-StreamThread-3] o.a.k.s.p.i.StreamThread [StreamThread.java:1453] - stream-thread [69d6d46a-3cb4-431e-9342-6305af1f73be-fa6a6ac0-8d20-403b-bd91-8c3497ada04f-StreamThread-3] Failed to commit stream task 1_0: 
org.apache.kafka.streams.errors.ProcessorStateException: task [1_0] Failed to flush state store idStore
	at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:262)
	at org.apache.kafka.streams.processor.internals.AbstractTask.flushState(AbstractTask.java:190)
	at org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:282)
	at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:264)
	at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:187)
	at org.apache.kafka.streams.processor.internals.StreamTask.commitImpl(StreamTask.java:259)
	at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:253)
	at org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:815)
	at org.apache.kafka.streams.processor.internals.StreamThread.access$2800(StreamThread.java:73)
	at org.apache.kafka.streams.processor.internals.StreamThread$2.apply(StreamThread.java:797)
	at org.apache.kafka.streams.processor.internals.StreamThread.performOnStreamTasks(StreamThread.java:1448)
	at org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:789)
	at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:778)
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:567)
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
Caused by: java.lang.NullPointerException: null
	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:847)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2975)
	at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:202)
	at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:201)
	at com.schibsted.spt.data.yggdrasil.common.serdes.JsonSerde$$anon$1.readValue(JsonSerde.scala:16)
	at com.schibsted.spt.data.yggdrasil.common.serdes.JsonSerde$JsonDeserializer.deserialize(JsonSerde.scala:35)
	at org.apache.kafka.streams.state.StateSerdes.valueFrom(StateSerdes.java:163)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.putAndMaybeForward(CachingKeyValueStore.java:90)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.access$000(CachingKeyValueStore.java:34)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore$1.apply(CachingKeyValueStore.java:78)
	at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:145)
	at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:103)
	at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:97)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:107)
	at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:260)
	... 14 common frames omitted

I've seen this failed to flush in production once as well, but I'm not sure where it comes from. Any ideas?

Forgot to mention that I'm here using a custom JsonSerde which uses Jackson below.

Emanuele Blanco · Answer 1 · Fri Sep 22 2017 04:02:33 GMT+0800 (China Standard Time)

Have you tried to ask on the Kafka ML? It doesn't seem to me that this could be related to the library use...

Mats Julian Olsen · Answer 2 · Mon Oct 02 2017 17:34:14 GMT+0800 (China Standard Time)

You're probably right, I think my way to test was a bit flawed. I've moved to using the KStreamTestDriver.