andrylavr / seslog

seslog is nginx syslog server. It collects nginx's access logs and write them to ClickHouse

seslog

seslog is nginx syslog server. It collects nginx's access logs and write them to ClickHouse DB.

Make

make

Run

build/seslog-server -logtostderr=true

Run build/seslog-server --help to see all options

Install as systemd service

sudo make install

Install to nginx

Add new log_format to your nginx config (http section)

log_format seslog_format '$body_bytes_sent\t$connections_active\t$connections_reading\t$connections_waiting\t$connections_writing\t$content_length\t$http_host\t$http_referer\t$http_user_agent\t$http_x_forwarded_for\t$remote_addr\t$request_method\t$request_time\t$request_uri\t$scheme\t$status\t$tcpinfo_rtt\t$tcpinfo_rttvar\t$time_local\t$upstream_cache_status\t$upstream_response_length\t$upstream_response_time\t$upstream_status\t$uri\t$sent_http_location';

Then add access_log (preferred section)

access_log syslog:server=<YOUR_SESLOG_IP>:5514,tag=<YOUR_PROJECT_NAME> seslog_format;

For example:

access_log syslog:server=127.0.0.1:5514,tag=php_admin_panel seslog_format if=$sesloggable;

Tips

Please use $loggable (or anything like that) variable to avoid useless logging

e.g. (http context):

map $request_uri $sesloggable {
    default                                             1;
    ~*\.(ico|css|js|gif|jpg|jpeg|png|svg|woff|ttf|eot)$ 0;
}

SELECT data

Select TOP-30 Referrer Domains

SELECT
    http_referer_domain,
    count() AS cnt
FROM access_log 
WHERE nginx_event_date = today()
GROUP BY http_referer_domain
ORDER BY cnt DESC
LIMIT 30