Plog is short for "Parse Log", it's a log-handle framework designed for handling and analyzing log file generated from Apache, nginx etc.
Inspired by FlumeNG, I divied the project into four parts:source,channel ,sink and common. and I have complete the core the project like the deal-interval-control,the thread-control ,also have completed some simple functions of source,channel and sink.
The following is a demo configuration file.
[source]
source_module=file_source
source_file=./test/plog_demo.log
source_interval=5
[channel]
channel_module=regrex_channel
channel_filter_regex=([\w\d.\s,]{0,})\s([0-9.]+)\s(\d+|-)\s(\w+)\s\[([^\[\]]+)\s\+\d+\]\s"((?:[^"]|\")+)"\s(\d{3})\s(\d+|-)\s"((?:[^"]|\")+|-)"\s"(.+|-)"\s"((?:[^"]|\")+)"\s"(.+|-)"$
channel_dict_key=domain_name,ip,response_time,TCP_status,date_time,request_url,response_code,size,ref,item1,agent,item2
[sink]
interval=60
sink_module=zabbix_sink
sink_service=cacheL2
sink_zabbix_send_file=/tmp/zabbix_send_info
sink_zabbix_sender=/usr/bin/zabbix_sender
sink_zabbix_conf=/etc/zabbix/zabbix_agentd.conf
[log_config]
logging_format=%(asctime)s %(filename)s [funcname:%(funcName)s] [line:%(lineno)d] %(levelname)s %(message)s
logging_level=20
logging_filename=/tmp/plog.log
I use ConfigParse to parse the configue file.
####The Part of Source In this part,we deal with the input ,its type may be file ,exec,TCP socket ,and so on ,but I don't care about the type of ur input,you just need to complete a python script,and its name depends on ur inteest,what you need is to write the name in plog.conf,like following:
source_module=self-define-script-name
you can read plog/source/youself_define_source.py to see what you should complete
####The Part of Channel In this part,we deal with parsing log into some groups of value.you also need to complete a python script,and its name also depends on your interest,what you need is to write the name in plog.conf,like following:
channel_module=filter_log
also,you can read plog/channel/youself_define_channel.py to get to know what you should complete
####The Part of Sink In this part,you also need to complete a python script,and its name also depends on your interest,what you need is to write the name in plog.conf,like following:
sink_module=cacheL2get_monitor
you can read plog/sink/youself_define_sink.py to get to know what you should complete
####How To Run A Demo 1.git clone https://github.com/xluren/Plog
2.cd ./Plog/test
3.nohup sh gen_log.sh &
4.cd .. && python plog.py -c plog.conf
5.you will see a file**/tmp/zabbix_send_info_cacheL2**,its contents like followings:
[xluren@test Plog]$ cat /tmp/zabbix_send_info_cacheL2
test.145 cacheL2_200 57.000000
In China,we name Nov 11 as "Single Day",but in the day of Nov 11 2014,I fix the bug of plog and also restruct it . So today I am very happy,event more than that I have a girlfriend :-)