Giters

Hurence / logisland

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

Home Page:https://logisland.github.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

add a Processor Pool Service in logisland core

oalam opened this issue · comments

commented

pipeline objects (aka processors) are intanciated at each micro-batch by spark executors. This leads to too many objects creation and GC purge.
we could instead lazily create a pool of processors into a CoreControllerService that handles a pool of objects needed for record processing.

may be with https://commons.apache.org/proper/commons-pool/