Generic mechanism to add metadata to Zeek events.

[Jan]

As discussed in #331, it would be nice to have a generic mechanism to add metadata to Zeek events. @awelzel sketched the following:

One more thought: Could we go for a a more extensible header-style field instead of the atomic field that is just a timestamp? That is, append a vector<vector<count, data>> where the count identifies what's contained.

Mainly, while the current idea is to only attach network time to an event, what if in the future more information is need or different approaches want to be evaluated? That would require another broker change and another round of appending-a-field-trying-to-stay-compatible. An extensible header format avoids that at the cost of extra complexity.

So the event headers (maybe there's a better name) would just be the following currently and populated up in Zeek land: Broker wouldn't actually need to know about about TIMESTAMP_NETWORK - that's a const up in Zeek land.

[
  [TIMESTAMP_NETWORK, double|uint64_t timestamp],
]

I would hope the extra marshaling / processing overhead is negligible compared to the involved IO and that encoding overhead is acceptable.

Some use cases @awelzel and @ckreibich brought up in the discussion:

• Propagating wallclock time of the event in addition to network time (latency analysis on an event level?)
• Ever increasing event clock or epoch from the sender (or even some form of logical event clock)
• Vector clocks of event clocks received from other nodes at the time of sending
• Name of originating node/package/script/context
• Troubleshooting info
• Things like transaction IDs that currently feel a bit clunky as explicit event arguments
• Protocol version info

And as Christian pointed out:

These bring up a question of how we might make event metadata accessible to the script layer, btw — something to think about.

This probably relates to #340.