Hurence / logisland

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

Home Page:https://logisland.github.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

add SplitRecord processor

oalam opened this issue · comments

this processor takes 1 record in and gives n records out according to dynamic parameters

example conf

    - processor: split_record
      component: com.hurence.logisland.processor.SplitRecord
      configuration:
        # default false
        keep.parent.record: false 
        # default false, if false the new record_type will be the name of the dynamic property
        keep.parent.record_type: false
        # default true, if false the new record_time will is set to processing_time
        keep.parent.record_time: false
        # dynamic parameters 
        record_type1: fieldA, fieldB
        record_type2: fieldC
        record_type3: fieldA, fieldD

will give

R(record_time0, record_type0, record_id0, fieldA, fieldB, fieldC, fieldD)
=>
R1(record_time0, record_type1, record_id1, parent_record_id =record_id0, fieldA, fieldB)
R2(record_time0, record_type2, record_id2, parent_record_id =record_id0, fieldC)
R2(record_time0, record_type3, record_id3, parent_record_id =record_id0, fieldA, fieldD)