logstash-plugins / logstash-patterns-core

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Implement ECS-Compatibility Mode

kares opened this issue · comments

this one falls under: logstash-plugins/logstash-filter-grok#157

There's going to be 2 set of patterns provided out-of-the-box that provide around the same functionality (at start).
The existing (legacy) set and an ECS set of patterns - with mostly captures reviewed/renamed for compliance.

Patterns are already logically split by functionality into separate files - good enough to convert the whole set one-by-one :

Post TODOs

  • ship (legacy) pattern updates 4.2.0 before ECS-ified release

  • check if ECS 1.6 is around with http.response.mime_type #270 (comment)
    in the mean time consider renaming squid.response.content_type to squid.response.mime_type ?
    UPDATE 1.7 shipped http.[request|response].mime_type as GA
    ✔️ renamed squid.response.content_type at 3b86557

  • host.name vs host.hostname
    https://github.com/logstash-plugins/logstash-patterns-core/pull/262/files#r459038278
    https://github.com/logstash-plugins/logstash-patterns-core/pull/262/files#r504850250
    ✔️ we'll be using hostname, users should opt-in to name (due SIEM)

  • introduce grok conversions for ES ingest node compatibility (strict :int and :float parsing modes, add :long, :double, :boolean + we could also use an AS style :bool conversion e.g. with bro's T/F flags)
    than review patterns
    ✔️ won't block shipping ECS-ified patterns - we should double check :int capture INT-like values
    ...:int captures reviewed 6dd657b

  • need to make sure to set event.original (in grok?) and potentially remove message field before reaching grok
    patterns that extract message to keep the field flat (or default to running with overwrite => [ 'message' ]) in ECS mode

  • type-casting doesn't always work due a grok library bug

  • avoid mixed <style> named captures (#265 (comment))