Elastic 'common schema' support?

[Ash A.]

Elastic.co has a 'common schema' that they encourage, and it'd be nice if the format was understood by jl.

Spec: https://www.elastic.co/guide/en/ecs/current/index.html

Sample line:

{
  "service": { "name": "gunicorn" },
  "@timestamp": "2020-10-23T03:35:49.324754+00:00",
  "message": "10.244.1.180 - - [23/Oct/2020:03:35:49 +0000] \"GET /users/users/notices/ HTTP/1.1\" 200 4942 \"http://localhost:4200/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36\"",
  "time": 1603424149.3247535,
  "log": {
    "level": "INFO",
    "logger": "gunicorn.access",
    "origin": {
      "file": { "line": 570, "name": "/app/ticketing/utils/log.py" },
      "function": "access"
    }
  },
  "process": {
    "pid": 17,
    "name": "MainProcess",
    "thread": { "name": "MainThread", "id": 140056871733056 }
  },
  "request": {
    "scheme": "https",
    "path": "/users/users/notices/",
    "method": "GET",
    "customer": "test",
    "view": {
      "args": [],
      "app": "users",
      "namespace": "users",
      "name": "users:user-notices"
    }
  },
  "customer": "test",
  "event": { "duration": 78518000 },
  "http": {
    "request": { "method": "GET", "referrer": "http://localhost:4200/" },
    "response": { "body": { "bytes": 4942 }, "status_code": "200" },
    "version": "1.1"
  },
  "related": { "ip": ["10.244.1.180"] },
  "source": { "address": "10.244.1.180" },
  "url": { "path": "/users/users/notices/", "query": "" },
  "user_agent": {
    "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36"
  }
}

[Koen Bollen]

Hey Ash,
Thank you for the suggestion, I agree: would be nice if jl would support this format.
I'll try to make some time soon.

[Koen Bollen]

I've created the following pull request: #20
This adds better support for nested fields and I've added some common fields of ECS in there.

It should now pickup the log.level correctly and it will allow you to select nested fields:

$ cat your-ecs.log | jl -f request.path

I did not touch the contents of the "message" field, since that would would be to specific for these kinds of gunicorn logs.

@dragonpaw Could you try this branch on your logs? Let me know if you need any help with that or expect any different behaviour.

[Ash A.]

Looks to be working great. (And yeah, don't wanna mess with 'message', it's way too specific.) There's a screenshot on the PR of it in action. Thank you!

[Koen Bollen]

I've merged the pullrequest, released jl and updated my homebrew tap (koenbollen/homebrew-public@71c5132).
Closing this issue now.
Thanks again for the suggestion 🎉