Logs with geo

Ingest and enrich logs with geoip information (country, city, latitude, longitude).

The original idea here was to enrich my apache logs for Loki so that I could use the Geomap view for my logs in Grafana.

Setup on the server

Get the GeoLite-City.mmdb from MaxMind GeoIP and store it to the geoip-db folder.
Install requirements as root python3 -m pip install -r requirements.txt because we'll run the crontab as root later for accessing apache logs

first run

Prepare two sample log files: sample-logs/access.log and sample-logs/other_vhosts_access.log

Attempt a first run like this

python3 main.py -o sample-logs/all_with_geoip.log -f sample-logs/access.log -f sample-logs/other_vhosts_access.log

cron

add this to roots crontab:

* * * * * /usr/bin/python3 /YOUR/PATH/logs-with-geo/main.py -o /var/log/apache2/all_with_geoip.log -f /var/log/apache2/access.log -f /var/log/apache2/other_vhosts_access.log

To run it more often than once a minute (eg every 10 seconds), you could use this:

*/1 * * * * /YOUR/PATH/logs-with-geo/runEvery.sh 5 "/usr/bin/python3 /YOUR/PATH/logs-with-geo/main.py -o /var/log/apache2/all_with_geoip.log -f /var/log/apache2/access.log -f /var/log/apache2/other_vhosts_access.log"

Watch all_with_geoip.log grow with added geo data (eg with tail -f all_with_geoip.log!

Simulate behaviour (manual testing)

prepare these files:

other_vhosts_access.log <- should be empty
other_vhosts_access.orig.log <- should contain some log lines (at least 100 for more fun)

run this in one window:

./log_faker.sh other_vhosts_access.orig.log other_vhosts_access.log

and this in another

while [ 1 ]; do
    python3 main.py -o sample-logs/all_with_geoip.log -f sample-logs/access.log -f sample-logs/other_vhosts_access.log
    sleep 3
done

Then watch the sample-logs/all_with_geoip.log grow with added geo data!

To Dos

lint with flake8
set up integration tests
make it log source agnostic (not only apache, but also nginx or even something else that logs ips)

ronnyworm / logs-with-geo