A Zeek log archival service.
This tool is derived from bro-atomic-rotate and intends to solve the same failings of Zeek's historical log-archival process: robustness and atomicity. It's rewritten with two further requirements in mind:
- Independence from ZeekControl. It's meant for use in conjunction with the upcoming Zeek Supervisor Framework.
- Independence from Python. In retrospect, it's dubious whether that's a benefit: implementing in C++ has little advantage over a solution done in a simpler Bash/Python script, so it may get completely rewritten later if any concrete maintenance burden/problems are encountered.
- CMake 3.0 or greater
- C++ compiler with C++17 support (GCC 7+ or Clang 4+)
- By default, compression is enabled by shelling out directly to
gzip
, either install that separately or disable/change the compression mechanism via the--compression=
flag.
Just run make
, which is short for:
$ mkdir -p build && cd build && cmake .. && make
Since zeek-archiver
is made for use with the Zeek Supervisor Framework,
you should first install Zeek and configure your Supervised Cluster, based
on the example given here which will rotate logs into $(cwd)/logger/log-queue/
:
https://docs.zeek.org/en/current/frameworks/supervisor.html#supervised-cluster-example
After, install/configure zeek-archiver
itself as a service:
$ make install
$ cp zeek-archiver.service /etc/systemd/system/
# Modify the ExecStart invocation in service file as needed.
$ systemctl enable zeek-archiver
$ systemctl start zeek-archiver
While zeek-archiver
is meant to be used with the Zeek Supervisor Framework,
it's still possible to use with ZeekControl in the time before it's entirely
succeeded by the Supervisor Framework. As an example of how to configure
zeek-archiver
to work with ZeekControl, add this code to your local.zeek
@if ( Cluster::local_node_type() == Cluster::LOGGER )
redef Log::default_rotation_dir = "/usr/local/zeek/logs/log-queue";
function my_rotation_format_func(ri: Log::RotationFmtInfo): Log::RotationPath
{
local open_str = strftime(Log::default_rotation_date_format, ri$open);
local close_str = strftime(Log::default_rotation_date_format, ri$close);
local base = fmt("%s__%s__%s__", ri$path, open_str, close_str);
local rval = Log::RotationPath($file_basename=base);
return rval;
}
redef Log::rotation_format_func = my_rotation_format_func;
redef Log::default_rotation_postprocessor_cmd = "";
@endif
Then run zeek-archiver
the same way as explained earlier to have it monitor
and rotate any logs that show up in /usr/local/zeek/logs/log-queue
.
The historical ZeekControl method for log rotation/archival looked like:
mv conn.log conn-yaddayadda.log
gzip < conn-yaddayadda.log > /bro/logs/2018/10/10/conn.09:00:00-10:00:00.gz
But that is not an "atomic" operation that's robust in the face of power less,
reboot, OOM, something trying to read .gz
files as they're created.
The archival process for each log also happened all concurrently with one
another, which creates problematic load spikes.
Instead, zeek-archiver
archives log files serially and atomically in a way
that depends on which criteria is met:
- If compression is desired:
gzip < src > dst.tmp && mv dst.tmp dst && rm src
- No compression, within same filesystem:
mv src dst
- No compression, across filesystems:
cp src dst.tmp && mv dst.tmp dst && rm src