Split messages

[Henrique Goncalves]

• I've read and understood the Contributing guidelines and have done my best effort to follow them.
• I've read and agree to the Code of Conduct.
• I've searched for any related issues and avoided creating a duplicate issue.

Description

Hello. I would like to known how I can split messages, or prevent go-audit from aggregating them into a single log line.

For example:

2017-06-29T00:46:11Z ip-10-0-0-99 go-audit[297]: {"sequence":2487,"timestamp":"1498697171.657","messages":[{"type":1300,"data":"arch=c000003e syscall=59 success=yes exit=0 a0=e4fee0 a1=e510a0 a2=e22540 a3=5a1 items=2 ppid=18824 pid=18848 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm=\"cat\" exe=\"/usr/bin/cat\" key=\"user_commands\""},{"type":1309,"data":"argc=2 a0=\"cat\" a1=\"/etc/passwd\""},{"type":1302,"data":"item=0 name=\"/usr/bin/cat\" inode=409296 dev=ca:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL"},{"type":1302,"data":"item=1 name=\"/lib64/ld-linux-x86-64.so.2\" inode=395436 dev=ca:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL"},{"type":1327,"data":"proctitle=636174002F6574632F706173737764"}],"uid_map":{"0":"root","1000":"henrique.goncalves"}}

Under "messages", there are several entries. This is proving a pain in the buttocks to parse with logstash. All my "split" tries didnt work.

How can I log each of those messages separated, even if duplicating the sequence and timestamp values, and the uid_map.

Reproducible in:

go-audit version: all
OS version(s): all

Steps to reproduce:

1. Log anything

Expected result:

Messages are logged separated

Actual result:

Messages are put into a JSON array somewhat hard to parse using logstash.

Attachments:

e.g. Logs, screenshots, screencast, sample project, funny gif, etc.

[Nate Brown]

The reason we built go-audit was to coalesce all the individual messages that make up an entire event as quick as possible. We use streamstash at Slack to make these events more meaningful to humans. You could certainly port the parsing code from streamstash into logstash.

I piped the line you provided through streamstash using this and the output is a much more elasticsearch friendly blob:

{
  "message": "henrique.goncalves as root succeeded to execve `cat /etc/passwd` via `/usr/bin/cat`",
  "go-audit": {
    "timestamp": "2017-06-29T00:46:11.657Z",
    "sequence": 2487,
    "unknown": [],
    "syscall": {
      "arch": {
        "bits": "64",
        "endianness": "little",
        "name": "x86_64"
      },
      "success": "yes",
      "exit": "0",
      "a0": "e4fee0",
      "a1": "e510a0",
      "a2": "e22540",
      "a3": "5a1",
      "items": "2",
      "ppid": "18824",
      "pid": "18848",
      "auid": {
        "name": "henrique.goncalves",
        "id": "1000"
      },
      "uid": {
        "name": "root",
        "id": "0"
      },
      "gid": "0",
      "euid": {
        "name": "root",
        "id": "0"
      },
      "suid": {
        "name": "root",
        "id": "0"
      },
      "fsuid": {
        "name": "root",
        "id": "0"
      },
      "egid": "0",
      "sgid": "0",
      "fsgid": "0",
      "tty": "pts0",
      "key": "user_commands",
      "id": "59",
      "session_id": "1",
      "name": "execve",
      "command": "cat",
      "executable": "/usr/bin/cat"
    },
    "paths": [
      {
        "name": "/usr/bin/cat",
        "inode": "409296",
        "dev": "ca:01",
        "mode": "0100755",
        "ouid": {
          "name": "root",
          "id": "0"
        },
        "ogid": "0",
        "rdev": "00:00",
        "nametype": "NORMAL"
      },
      {
        "name": "/lib64/ld-linux-x86-64.so.2",
        "inode": "395436",
        "dev": "ca:01",
        "mode": "0100755",
        "ouid": {
          "name": "root",
          "id": "0"
        },
        "ogid": "0",
        "rdev": "00:00",
        "nametype": "NORMAL"
      }
    ],
    "execve": {
      "command": "cat /etc/passwd"
    },
    "proctitle": "cat /etc/passwd"
  }
}

[Ryan Huber]

Closed due to inactivity. I think nate explained well ^