- Setup virtual environment
python3 -m venv venv
- Install dependencies
pip install -r requirements.txt
- Run
httpm
python3 httpm.py [<OPTIONS>] PATH_TO_LOG
With extra configuration:
python3 httpm.py test.log -th 2 -adi 30 --traffic-stats response_codes request
_size
Run httpm
with the --help
flag to see which options are configurable (such as the high traffic threshold, statistics delay, and alert delay).
To test the log monitor, I wrote a script to simulate a live http access log using the sample_csv.txt
file which was provided to me.
To run the simulation:
- Run the log simulation script
python3 logger.py
- In another shell, run
httpm
python3 httpm.py test.log
There are five main classes in my design.
LogConsumer
is responsible for reading in lines from the target log file and writing them to the LogKeep
.
LogKeep
is a container for the log lines. It only holds log lines that have been read within the last
alert_delay_interval
seconds.
Alerter
is responsible for determining if any alert was triggered based on the log lines from the last alert_delay_interval
seconds.
The TrafficStatistic
classes are responsible for calculating different statistics based on the log lines from the last stats_delay_interval
seconds. Adding a new statistic is easy: just implement the TrafficStatistic
interface.
Monitor
represents the system as a whole. It uses the classes described above to implement the functionality of an http access log monitor.
LogConsumer
reads new lines from the log file and adds them toLogKeep
.- Every
stats_delay_interval
seconds,Monitor
reads the loglines from the laststats_delay_interval
seconds to calculate statistics and output them. - Every
alert_delay_interval
seconds,Monitor
outputs an alert if one was triggered within the lastalert_delay_interval
seconds.
NOTE:
Monitor
continuously checks if an alert was triggered, but only outputs an alert to the console at every interval. The time posted in the alert is the time at which the alert was triggered, not the current time (when the alert is output to the console).
Test are written with unittest
. To run:
python3 -m unittest tests.py
-
Better organization of files into submodules (folders...)
-
One possible improvement could be to use filesystem events to notify the monitor when new lines are written to the log file, instead of continuously polling it.
-
In terms of scalability, I would be interested in possibly expanding on the
LogKeep
idea, but using something likeRabbitMQ
instead. I am not sure, however, if this is overkill for a simple log monitor program. -
Have more statistics.
-
Have different alerts (e.g. several requests originating from the same ip? Lots of
500
response codes?, etc.) -
Notifications for alerts (email, sms?)
-
Actions on alert (e.g. make a request to some other server in response to certain alerts)
-
Nice console banner, colors, or even a gui