Giters

logstash-plugins / logstash-patterns-core

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

CLOUDFRONT_ACCESS_LOG pattern fails to match CloudFront logs

pa-jberanek opened this issue · comments

commented

Logstash information:

  1. Logstash version (e.g. bin/logstash --version) - 7.14.0
  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) - RPM
  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) - systemd
  4. How was the Logstash Plugin installed - n/a

JVM (e.g. java -version):

Bundled JVM

OS version (uname -a if on a Unix-like system):

Oracle Linux 8

Description of the problem including expected versus actual behavior:

The CLOUDFRONT_ACCESS_LOG pattern fails to match CloudFront logs, because the "x_edge_location" doesn't match with WORD as it (often) contains dashes, e.g.

LHR62-C3

Steps to reproduce:

  1. Build a pipeline to ingest CloudFront logs utilising a grok with the CLOUDFRONT_ACCESS_LOGS pattern
  2. Run the pipeline

Provide logs (if relevant):

commented

My fix was to take the pattern and create a version using DATA instead.

commented

Hey John, in order to speed up a fix for this issue it would have been nice to have a sample log line that fails the match ...

commented

OK, here's a line:

#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end
2021-08-24      00:24:40        LHR62-C3        33517   82.44.60.119    GET     d1236u0ikuk2zt.cloudfront.net   /p/101/thumbnail/entry_id/0_50xpj7v0/width/290/height/150/type/3        200     https://www.liverpoolfc.com/    Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2014_7_1%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/14.1.2%20Mobile/15E148%20Safari/604.1     -       -       Hit     YoIRNxF4o0fam7eNcIJ_QG24jMjjMNBvWK0xoveWisgYoWVzvyYFvQ==        open.http.mp.streamamg.com      https   289     0.003   -       TLSv1.3 TLS_AES_128_GCM_SHA256  Hit     HTTP/2.0        -       -       54902   0.003   Hit     image/jpeg      33046   -       -