___ ___ ___ /\ \ /\ \ /\__\ /::\ \ /::\ \ /:/ _/_ /:/\:\__\ /:/\:\__\ /:/ /\ \ /:/ /:/ //:/ /:/ /___ ___ /:/ /::\ \ /:/_/:/ //:/_/:/ //\ \ /\__\/:/_/:/\:\__\ \:\/:/ / \:\/:/ / \:\ \ /:/ /\:\/:/ /:/ / \::/__/ \::/__/ \:\ /:/ / \::/ /:/ / \:\ \ \:\ \ \:\/:/ / \/_/:/ / \:\__\ \:\__\ \::/ / /:/ / \/__/ \/__/ \/__/ \/__/
Probabilistic Production Log Summarizer
This program simplifies your production logs using a combination of probabilistic and exact near(est)-neighbor clustering techniques. The goal is to automagically produce a very good representative subset of your log even with important volume of data.
Usage example:
python main.py --fuzzy_fields log_message,100 --exact_fields severity --similarity_threshold 0.89 --logfile log_samples/error_log --log_format errorlog_apache
Supported log formats:
With a bit of plugin coding, it is posible to analyse virtually any kind of textual logs (see exemple at the end of this page). But out of the box, we support:
- Webserver access logs : apache/nginx
- Application server : weblogic
- syslog
Dependencies:
The following libraries are required to run PPLS
- python-Levenshtein
- redis (optional)
- IPython [notebook] (optional)
References
The algorithms used in this project are mainly inspired from chapter 3 of the excellent book: [Mining of Massive Datasets] (http://infolab.stanford.edu/~ullman/mmds.html)