dustalov / ballcone

Ballcone is a fast and lightweight server-side Web analytics solution.

Ballcone

Ballcone is a fast and lightweight server-side Web analytics solution. It requires no JavaScript on your website.

Screenshots

Design Goals

• Simplicity. Ballcone requires almost zero set-up as it prefers convention over configuration
• Efficiency. Ballcone performs lightning-fast analytic queries over data thanks to the underlying columnar database
• Specificity. Ballcone aims at providing visual insights on the HTTP access logs with no bloat

Features

• No JavaScript snippets required
• GeoIP mapping with the GeoLite2 database
• Extraction of platform and browser information from User-Agent

Architecture

Ballcone captures the access_log entries exported in JSON by nginx via the bundled syslog logger (65140/udp). These entries are stored in the embedded DuckDB database. Ballcone uses it to perform data manipulation and analytic queries. Also, Ballcone provides a convenient Web interface (8080/tcp) for accessing and observing the gathered data.

          +-----------+            +------------+
   HTTP   |           |   syslog   |            |   HTTP
<-------->+   nginx   +----------->+  Ballcone  +<-------->
          |           |    JSON    |            |
          +-----------+            +------------+
                                   |   DuckDB   |
                                   +------------+

For better performance, Ballcone inserts data in batches, committing them to DuckDB every few seconds (five seconds by default).

Requirements

• Python 3.9
• DuckDB ≥ 0.4.0
• nginx ≥ 1.7.1

Demo

This repository contains an example configuration of nginx and Ballcone. Just run the container from Docker Hub or build it locally. nginx will be available at http://127.0.0.1:8888/ and Ballcone will be available at http://127.0.0.1:8080/.

docker-compose up
# or
docker run --rm -p '127.0.0.1:8888:80' -p '127.0.0.1:8080:8080' dustalov/ballcone:demo

Naming and Meaning

Ballcone has two meanings.

First, it is the romanization of the Russian word балкон that means a balcony. You go to the balcony to breath some fresh air and look down at the things outside.

Second, if a ball is inscribed in a cone, it resembles the all-seeing eye (help wanted: dustalov/ballcone#8).

Regardless of the meaning you prefer, Ballcone helps you to watch your websites.

Installation

The simplest way to get started is to run make pipenv after cloning the repository. Just make sure Pipenv is installed.

Getting Ballcone

Running the Docker image is the simplest way to get started. Docker Hub contains automated builds of the Ballcone source code from GitHub: https://hub.docker.com/r/dustalov/ballcone. The following command runs Ballcone on 127.0.0.1: the syslog protocol will be available via 65140/udp, the Web interface will be available via 8080/tcp, and the data will be stored in the /var/lib/ballcone directory on the host machine.

docker run -p '127.0.0.1:8080:8080' -p '127.0.0.1:65140:65140/udp' -v '/var/lib/ballcone:/usr/src/app/duckdb' --restart=unless-stopped dustalov/ballcone ballcone -sh '0.0.0.0' -wh '0.0.0.0' -d 'duckdb/ballcone.duckdb'

However, Docker is not the only option. Alternatively, Ballcone can be packaged into a standalone executable using PyInstaller and runned as a systemd service (see ballcone.service as an example):

make pyinstaller
sudo make install-systemd
sudo systemctl start ballcone

Finally, Ballcone can be installed directly on the host machine for manual runs:

pip3 install -e git+https://github.com/dustalov/ballcone@master#egg=ballcone

Note that ballcone without arguments creates the ballcone.duckdb database file inside the current directory.

Configuring nginx

You need to define the JSON-compatible log format for your service in the nginx configuration file. Let us call it ballcone_json_example. This format is similar to the one used in Matomo (see matomo-log-analytics). It should be put before the server context.

log_format ballcone_json_example escape=json
    '{'
    '"service": "example", '
    '"ip": "$remote_addr", '
    '"host": "$host", '
    '"path": "$request_uri", '
    '"status": "$status", '
    '"referrer": "$http_referer", '
    '"user_agent": "$http_user_agent", '
    '"length": $bytes_sent, '
    '"generation_time_milli": $request_time, '
    '"date": "$time_iso8601"'
    '}';

Then, you should put this access_log directive inside the server context to transfer logs via the syslog protocol.

access_log syslog:server=127.0.0.1:65140 ballcone_json_example;

Please look at the complete example of nginx configuration in demo/nginx.conf.

Roadmap

Roadmap is available at https://github.com/dustalov/ballcone/issues.

Alternatives

• Web analytics solutions: Matomo, Google Analytics, Yandex.Metrica, etc.
• Columnar data storages: ClickHouse, Citus, MariaDB ColumnStore, etc.
• Log management: Graylog, Fluentd, Elasticsearch, etc.

Copyright

Copyright © 2020–2023 Dmitry Ustalov. See LICENSE for details.