heka-stream-aggregator
Stream aggregation plugin for Mozilla Heka
StreamAggregatorFilter
This is a filter that can be used for aggregating multiple payloads into a single message. It accepts an encoder option that will be used prior to aggregating each payload. Each payload is separated by the delimiter config value.
Config:
-
stream_aggregator_tag: Optional tagging for identifying new pack down the pipeline since you will lose any Fields previously held. This setting creates a new Heka Field called "StreamAggregatorTag" and is given the value of this option. Defaults to "aggregated"
-
flush_interval: Interval at which accumulated payloads should be compressed in milliseconds. Defaults to 1000 (i.e. one second)
-
flush_bytes: Number of payloads that, if processed, will trigger them to be compressed. Defaults to 10.
-
encoder: This option will run each Payload through the specified encoder prior to aggregating.(required)
Example:
[filter_stream_aggregator]
type = "StreamAggregatorFilter"
message_matcher = "Fields[decoded] == 'True'"
stream_aggregator_tag = "aggregated"
flush_interval = 30000
flush_bytes = 1000000
encoder = "encoder_json"
delimiter = "\n" # Default
Example2 for bulk ES inserts: [filter_stream_aggregator] type = "StreamAggregatorFilter" message_matcher = "Fields[decoded] == 'True'" stream_aggregator_tag = "aggregated" flush_interval = 30000 flush_bytes = 1000000 encoder = "ESLogstashV0Encoder" delimiter = ""
[ESLogstashV0Encoder]
index = "logstash-%{program}-%{2006.01.02}"
type_name = "%{program}"
es_index_from_timestamp = true
id = "%{id}"
ES Output heka instance(don't use StreamSplitterDecoder):
[HttpOutput]
message_matcher = "Fields[decoded] == 'True'"
address = "http://es01.foo.bar:9200/_bulk"
encoder = "encoder_payload"
StreamSplitterDecoder
Used inside of MultiDecoder for splitting a payload generated by a StreamAggregatorFilter
Example: [multi_decoder] type = "MultiDecoder" order = ['zlib_decoder', 'split_decoder', 'json_decoder']
[multi_decoder.subs.zlib_decoder]
type = "ZlibDecoder"
[multi_decoder.subs.split_decoder]
type = "StreamSplitterDecoder"
delimiter = "\n"
[multi_decoder.subs.json_decoder]
type = "SandboxDecoder"
script_type = "lua"
filename = "/usr/share/heka/lua_decoders/json_decoder.lua"
preserve_data = true
To Build
See Building hekad with External Plugins for compiling in plugins.
Edit cmake/plugin_loader.cmake file and add
add_external_plugin(git https://github.com/michaelgibson/heka-stream-aggregator master)
Build Heka: . ./build.sh