LOGS NOTIFICATION TOOL - logNOT
Abstract
logNOT is application that implements simple tool that can be
instructed to track and measure frequency of textual log items,
and to trigger specific action/command when ever frequency of
monitored log item or items goes above or bellow defined
frequency limits. Frequency is defined as number of item
occurrence per time period. logNOT can process logs in “real-time”
at moment immediately after logs are writhed to log file. Action
or command that will be triggered by logNOT is defined by user,
can be any executable program or shell script. What this action
will do depends on user. Other words it is building block of
larger system implementation, and its goal is to simplify that
implementation, removing one of the problems that disrupt big
picture. logNOT is very lightweight application that dose not add
any significant payload to system.
Table of Contents
0.1 Introduction
0.2 Important requirements and dependencies
0.3 Features
0.4 Installation
0.4.1 Running
0.5 Log file monitoring
0.6 Logs filtering
0.6.1 Log file permissions
0.7 logNOT configuration
0.7.1 Command line options
Usage:
0.7.2 Configuration file
0.7.3 Reloading configuration
0.8 Summary
0.9 Examples
0.9.1 Notifying on system errors
0.9.2 SSH brute force attack detection
0.9.3 Log size notification
0.1 Introduction
Motivation behind this project is to get simple reliable way of
fetching logs of interest at “real-time” for monitoring or
indexing purpose. Concept of real-time here is “soft real-time”,
there is no guarantee that logNOT will respond within strict
deadline, but in most cases it will respond in hundreds of
second, except in cases of extra ordinary load which can cause
such slow response of all applications. Purpose of log monitoring
is ability to define and extract certain events of interest from
the system. Such as brute force attack detection on service,
detection of possible system problems, and so on. Once this is
done, left to do is handling this events, like providing
notifications or any action required. The other possible aspect
of logNOT is to serve as middle connection between logs and
indexing service. logNOT already knows to track and extract
events from logs. The other part of logNOT can be implementation
of ability to pass all and/or filtered logs to specific service
that will serve for further analyser of log content. But this is
not primary goal for initial launch of this project.
0.2 Important requirements and dependencies
• logNOT is designed and implemented only to work on Linux
systems
• libpcre 8.02 or higher (Perl Compatible Regular Expressions C
library)
• Linux kernel 2.6.18 or higher
• INOTIFY feature enabled in Linux kernel.
Symbol: INOTIFY [=y]
Prompt: Inotify file change notification support
Defined at fs/notify/inotify/Kconfig:1
Location:
-> File systems
Selected by: AUDIT_TREE [=n] && AUDITSYSCALL [=n]
Symbol: INOTIFY_USER [=y]
Prompt: Inotify support for userspace
Defined at fs/notify/inotify/Kconfig:16
Location:
-> File systems
Selects: FSNOTIFY [=y]
0.3 Features
• Real-time monitoring of log files.
• Down-bound and and up-bound frequency of logs notification.
• Log file size notification.
• Filtering logs with PCRE (Perl Compatible Regular Expressions).
• Frequency count by CRC (Cycle Redundancy Check) of logs match.
(Meaning explained later)
• Automatic handling of logrotate. There is no need to configure
logrotate to send SIGHUP signal to logNOT for files monitored
by logNOT. Actually, if you send SIGHUP signal to logNOT it
will only reopen its own log file.
0.4 Installation
Currently logNOT is distributed as source code package that can
be fetched from https://github.com/fkasumovic/logNOT. After
fetching logNOT package it must be extracted
~# tar zxf lognot-0.1.tar.gz
After extraction with above command in current working directory
will be created lognot-0.1 directory. Change directory to newly
extracted directory:
~# cd lognot-0.1
Execute autogen.sh script in directory:
~# ./autogen.sh
Next step is to run configure script:
~# ./configure --prefix=/usr
For debug version run configure following way:
~# ./configure --prefix=/usr --enable-debug
To include additional debug features as dead lock detection and
additional logs, export CXXFLAGS=” -D_DEBUG” variable:
~# export CXXFLAGS=" -D_DEBUG"; ./configure --enable-debug
In order to compile run:
~# make all
And as final step install logNOT by executing:
~# make install
0.4.1 Running
To run logNOT you basically need to create configuration file
that will be specified. Sample configuration file 'sample.conf'
is distributed with source code. By default logNOT will execute
in foreground and write its output to syslog as in following
example:
~# lognot -c sample.conf
By default, if configuration file is not specified logNOT will
search for it at /etc/lognot.conf, if configuration file is not
found logNOT will exit with error. If you want logNOT to write on
standard output specify -s flag as following:
~# lognot -c sample.conf -s
If you want to specify log file where logNOT should write its
logs use -l <file path> flag. And finally to run logNOT as daemon
specify -d flag:
~# lognot -d -c sample.conf
Above will run logNOT as background process which will use
sample.conf as configuration and write its own logs to syslog.
More options like running under specific user account and so on
is explained in logNOT configuration section of this document. If
logNOT hangs at startup, check the log or run it with -s flag to
see error message.
0.5 Log file monitoring
logNOT will perform “real-time” monitoring from moment it opens
log file for reading. It will position at end of log file
ignoring all logs from history. When ever content of file
changes, logNOT will read all new lines till end of last
completed line, and process each of extracted lines. If there is
no changes logNOT will not execute at all. It will not do
periodic checks for changes as similar tools do, rather it will
wait for kernel notification about current state of file. This
cycle will repeat for all time logNOT is running. History in the
logs that appears in file before logNOT starts to monitor it, are
not subject of interest, monitoring in this case is about knowing
what is going on at presence, what was happen before is subject
of logs history analysis. Mixing this two leads to logical
issues, and presents miss approach to problem.
0.6 Logs filtering
Detecting specific log items in the log is defined using perl
compatible regular expressions. Any log item that match to
specified regular expression will be counted. More than one
criteria are allowed to be defined per single log. logNOT is
configured trough ini configuration, each log item that need to
be monitored is specified by ini section, multiple ini section
can refer to single log, that way you can monitor multiple log
items per single log, and define different behaviour for each of
them. Following example represents configuration for ssh brute
force attack detection.
[SSH_AUTH]
regex=error: PAM: Authentication failure for illegal user (.*)
from (.*)$
path = /var/log/messages
usecrc=TRUE
upbound_action = /root/block_source "$2"
upbound_freq=5/30
[SSH_IU]
regex=\]: Invalid user (.*) from (.*)$
path = /var/log/messages
usecrc=TRUE
upbound_action = /root/block_source "$2"
upbound_freq=5/30
Meaning of above example is that if there is more then 5 failed
authentication attempts within 30 seconds then, call the script
that will block source ip. Another example would be to define
upbound_action to block source address matched from log item, by
calling iptables directly.
upbound_action=/sbin/iptables -I SSHD -s $2 -j DROP
In above example:
regex parameter defines filter for logs by specifying perl
regular expression
upbound_freq defines how many times log must repeat in some
time period, in format count/time in seconds
path defines monitored log file location
upbound_action specifies command to execute on matching
criteria regex and upbound_freq
usecrc will cause logNOT to trigger notification only if log
repeats more then 5 times from same IP
logNOT have ability to pass regular expression matches from
matched log item in form $#, where $1 is first match $2 second
and so on... Above expression is defined to extract ip for which
asterisk was reported, failed registration. If you need to use $
character in action specification escape it with \ character as
'\$', if you need to specify \ character specify it as '\\'. In
general case, there can be multiple different messages that match
to same regular expression, that's why logNOT takes crc of each
match and counts per this crc. For example, if you have many
users and all they authenticate at same time it will not trigger
logNOT if 6 different users miss type password or username. But
if you have more then 5 authentication attempts failed from same
ip address in 30 seconds that will trigger logNOT. Cause of this,
special care must be taken when specifying expression with usecrc
option enabled, if it includes date time part of message for
example that will change for almost each new log item, it may not
work as expected to work. Almost any log includes date and time,
but its easy to avoid this issue, cause logNOT will not crc whole
log item but part of it that is exact match to what is specified
by match_expr. Specifying $@ will pass complete log item, this
can be used for passing filtered logs (or even all logs) to
another application if required. logNOT also allows to specify
down bound frequency, that will trigger action if specified log
item is less frequent then downbound_freq. Reason behind this is
that you may expect logs from some applications on regular base,
and if they are not present would mean that something is wrong
and some action should be taken in that case. Setting 0/0 for
frequency, either for up bound or down bound frequency disables
frequency detection and notification will be triggered on every
match.
0.6.1 Log file permissions
logNOT requires read permissions on files it monitors, and read
permissions on directory where log file is placed if log will be
rotated. Cause it is using inotify Linux kernel feature that can
only watch inode if you have read permissions on it.
0.7 logNOT configuration
There is two types of configuration parameters to logNOT, and
each is used for appropriate options. There are command line
options to start logNOT with, and configuration file.
0.7.1 Command line options
Usage:
lognot --config <filename> [options] lognot --test <regexp>
Options:
-h --help Display this usage information and
exit.
-c --config <filename> This is configuration file location
(required).
-l --logfile <filename> Write logs in to this file.
-d --daemon Start as background application.
-s --stdout Write logs to standard output.
-u --uid <uid> Change uid
-g --gid <gid> Change gid
-w --chdir <directory> Change working directory
-p --pid <full path> Set process pid file path. Applies
with --daemon only.
-t --test <filename> Test specified configuration file.
-T --retest <reg expr> Test specified regular expression
on line/s from standard input.
-v --verbose Run as debug.
(Careful! frequent logs)
-V --version Print version information and exit.
0.7.2 Configuration file
As already shown, logNOT configuration is simple ini file. Each
section of ini file defines single log item to monitor, and as we
stated before multiple items can refer to same log file. Meaning
of each configuration file options is described in following
list.
[general] Is a special section, meant to be used for global
configuration parameters and default configuration values.
Following are global options which apply only to [general]
section:
athread_count Specify number of action threads. logNOT notifies
by executing specified action. Each action is first pushed to
execution queue, actions pending in the queue are fetched by
action threads and executed. If there is only one thread and
multiple actions triggered at same time, actions would be
executed sequentially one by one. If there is more then one
action thread, then more then one action can be triggered at
same time. Default value is 2.
uid Allows you to specify effective user id for logNOT process.
Command line option overrides configuration file if specified.
gid Allows you to specify effective group id for logNOT
process. Command line option overrides configuration file if
specified.
All other options can be specified in [general] section, and they
will stand for default in all other log items sections where this
options are not specified. logNOT is configuration is organized
per monitored items, each new item creates as new section in
configuration file [item name]. Bellow list of options can be
specified per each item:
path This option is to specify location of monitored file. This
options is required, if not specified logNOT will fail at start
up with error.
regex Specify here with perl regular expression what are the
logs of interest, if you want all logs simple set .* for the
value, or don't specify at all cause default value is .* . For
more options please refer to PCRE documentation, and examples.
upbound_action Is to specify what should be executed, other
words specify how to notify about watched log events. It allows
to specify shell command which will be executed by calling
system() function from C standard library. This option also
allows to pass regular expression matches to specified shell
commands, respectively $0 stands for match 0, $1 for match 1,
and so on in array of matches from regular expression (refer to
PCRE documentation for more information). If you want to pass
complete current log line where event triggered with use $@. If
$ character is otherwise used and not to specify argument it
should be escaped with backslash \ escape.
upbound_freq If not specified upbound_action will be executed
on each match. Otherwise it defines how many times monitored
log item ca repeat in some time period specified in seconds
units before logNOT will execute up bound action. Frequency is
specified in format <count>/<time seconds>.
downbound_action Is equivalent to upbound_action, with
difference its executed in case monitored item frequency drops
bellow value specified with downbound_freq option and it dose
not accept arguments from regular expression match, simple
cause it actives when no logs that match.
downbound_freq Define down limit for frequency expected for
monitored log item to appear. If not specified or specified as
0/<time> option is disabled.
usecrc This option accepts 0,1,yes,no,true or false values, and
is case insensitive. If enabled it will tell logNOT to measure
frequency per CRC (Cycle Redundancy Check) of $0 match. This is
useful if you want to trigger action only if repeated logs are
exactly same.
size Specify size of file on which if reached logNOT needs to
notify by executing command specified by size_action. logNOT
will also notify when ever size of file reaches 2xsize, 3xsize
and so on, on each value which is multiple of size. This is not
exact accurate on level of single byte, especially for small
values of size, simple cause writhed data at once can reach
multiple of size in single event and in that case logNOT will
notify only once. Its easy to check, simple set size to 1B (1
byte) and write some text line to monitored file it will
definitely be more then one byte. Value is specified as
<number><unit>, where unit can be B - for bytes, K - for
kilobytes, M - for megabytes and only on 64 bit systems G - for
gigabytes. If unit not specified but only number default unit
is byte.
size_action Allows to define action to be executed when ever
monitored file reaches multiple of size value option.
0.7.3 Reloading configuration
There is situations when its useful to change configuration of
logNOT with shortest interrupt possible on its current
monitoring. Sending SIGUSR1 signal to logNOT will cause it to
reload its configuration file. If there is errors in
configuration, logNOT will log errors and continue with original
configuration without interrupt. If everything is ok with new
configuration, logNOT will reinitialize to monitor per new
configuration settings. Always check if configuration is
successfully reloaded after making changes.
0.8 Summary
logNOT is meant to be simple, easy to manage and control. logNOT
can be deployed as stand alone application for monitoring logs,
and also can be easily integrated as part of bigger system.
logNOT can monitor for occurrence of specific log item/s,
specified by regular expressions, and allows to specify any
desired log item frequency as additional condition for firing
action. And logNOT provides mechanism for filtering, extracting
and passing information from logs, and as well passing of
complete log items.
0.9 Examples
0.9.1 Notifying on system errors
This is simplest example used to detect any logs that contains
word error. Following is content of logNOT configuration file.
[messages]
path=/var/log/messages
regex=error
upbound_action = /root/scripts/email_warning $@
/root/scripts/email_warning is a shell script you can create
using sendmail or similar tools for emailing warning message
about potential issues on the system. One example would be
following:
#!/bin/bash
cat | sendmail -t <<EOF
from: system1@domainname
to: user@domainname
subject: System Warning
There is following error message in the log:
$@
EOF
exit $?
0.9.2 SSH brute force attack detection
Following example demonstrate how to block source ip for 15
minutes if ssh authentication fails more then 3 times in one
minute.
[SSH_AUTH]
regex=error: PAM: Authentication failure for (.*) from (.*)$
path = /var/log/messages
usecrc=TRUE
upbound_action = /root/block_source "$2"
upbound_freq=3/60
[SSH_IU]
regex=\]: Invalid user (.*) from (.*)$
path = /var/log/messages
usecrc=TRUE
upbound_action = /root/block_source "$2"
upbound_freq=3/60
Script block_source used above as part of upbound_action can be
something like following:
#!/bin/bash
if [ -z "$1" ];then
# script requires argument
exit 0
fi
# log can contain domain name instead ip address and we want to
resolve to ip
# it dosen't metter if ping replies or not
ip=`ping -c 1 -i 0.1 $1 | pcregrep "^PING" | awk '{ print $3 }'`
ip=${ip##\(} ip=${ip%%\)}
#make log about this action and block source ip
logger -t "ssh_bf_detection" "BLOCKING IP: $ip"
iptables -I INPUT -s $ip -j DROP
# unblock ip after 15 minutes
chain=`iptables -n --line-numbers -L INPUT | grep "$ip" | awk
'{print $1}'`
echo "iptables -D INPUT $chain" | at now + 15 minutes
exit 0
0.9.3 Log size notification
In this example we simple track size of log and we want to email
warning message if log size goes above 100 megabytes:
[messages]
path=/var/log/messages
size=100M
size_action=/root/scripts/email_warning “Message about log size”