hilt86 / zeek_to_parquet

Simple python script to convert Zeek ascii logs to parquet format and upload to Amazon S3

zeek_to_parquet

Simple python script to convert Zeek ascii logs to parquet format and upload to Amazon S3

Installation

Install dependencies

pip install zat awscli fastparquet s3fs

Add your credentials

aws configure

Edit /opt/zeek/share/zeekctl/scripts/archive-log script (let me know if there is a better way). I've placed this script just before archive-log deletes the logfile.

158
159 # convert zeek logs to parquet and uploda to s3
160 /usr/bin/python3 /root/to_parquet.py $file_name $base_name s3://zeek.threatbear.co/
161
162 rm -f $file_name

NOTE : Make sure you put the trailing slash on the s3://zeek.threatbear.co/ URL otherwise the script will fail.

IMPORTANT: Zeek package updates will overwrite /opt/zeek/share/zeekctl/scripts/archive-log so until we find another way of running zeek postprocessors YOU WILL NEED TO MAKE THE EDIT AFTER EACH UPDATE!!

Restart Zeek zeekctl restart

IAM Policy Example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::zeek.threatbear.co/*"
        }
    ]
}