Implement ECS-Compatibility Mode
kares opened this issue · comments
this one falls under: logstash-plugins/logstash-filter-grok#157
There's going to be 2 set of patterns provided out-of-the-box that provide around the same functionality (at start).
The existing (legacy) set and an ECS set of patterns - with mostly captures reviewed/renamed for compliance.
Patterns are already logically split by functionality into separate files - good enough to convert the whole set one-by-one :
- aws #287
- bacula #295
- bind #281
- bro #286
- exim #283 (base-line) #290
- firewalls #293 and #296
- grok-patterns - core set #262
- haproxy #289
- httpd #280 #267
- java #268
- junos #294
- linux-syslog #262
-
maven - mcollective+mcollective-patterns #271
- mongodb #269
- nagios #276
- postgresql #266
- rails #265
- redis #263
- ruby #265
- squid #270
Post TODOs
-
ship (legacy) pattern updates 4.2.0 before ECS-ified release
-
check if ECS 1.6 is around with
http.response.mime_type
#270 (comment)
in the mean time consider renamingsquid.response.content_type
tosquid.response.mime_type
?
UPDATE 1.7 shippedhttp.[request|response].mime_type
as GA
✔️ renamedsquid.response.content_type
at 3b86557 -
host.name
vshost.hostname
https://github.com/logstash-plugins/logstash-patterns-core/pull/262/files#r459038278
https://github.com/logstash-plugins/logstash-patterns-core/pull/262/files#r504850250
✔️ we'll be usinghostname
, users should opt-in toname
(due SIEM) -
introduce grok conversions for ES ingest node compatibility (strict
:int
and:float
parsing modes, add:long
,:double
,:boolean
+ we could also use an AS style:bool
conversion e.g. with bro'sT
/F
flags)
than review patterns
✔️ won't block shipping ECS-ified patterns - we should double check :int capture INT-like values
✅...:int
captures reviewed 6dd657b -
need to make sure to set
event.original
(in grok?) and potentially removemessage
field before reaching grok
patterns that extractmessage
to keep the field flat (or default to running withoverwrite => [ 'message' ]
) in ECS mode -
type-casting doesn't always work due a grok library bug
-
avoid mixed <style> named captures (#265 (comment))