nomnomlog tails one or more log files and sends syslog messages to a remote central syslog server. It generates packets itself, ignoring the system syslog daemon, so its configuration doesn't affect system-wide logging.
Use Benefits:
- Cross platform, os service ready, multi-architecture single solution
- Collecting logs from servers & daemons which don't natively support syslog
- Filtering for or eliminating specific log messages and / or files using regular expressions.
- When reconfiguring the system logger is less convenient than a purpose-built daemon (e.g., automated app deployments)
- Aggregating files not generated by daemons (e.g., package manager logs)
Nomnomlog is a feature enhanced fork of the go-lang remote_syslog2. Nomnomlog differences include: different filter criteria prior to sending
logs to the central server, man pages, enhanced developer support and testing, more chip architectures, IPC Signals, and general fixes.
Prepackaged distributions are available on the nomnomlog releases page.
# For Debian/Ubuntu you might want the .deb package
sudo dpkg -i nomnomlog_0.1.0_amd64.deb
# For RHEL/Centos you might want the .rpm package
sudo rpm -i nomnomlog-0.1.0-1.x86_64.rpmPrecompiled binaries in tar/gz for generic Linux are also available.
Untar the package, copy the "nomnomlog" executable into your $PATH, and then customize the included example_config.yml with the log file paths to read and the host/port to log to.
Optionally, move and rename the configuration file to /etc/nomnomlog-config.yml so
that nomnomlog picks it up automatically. For example:
sudo cp ./nomnomlog /usr/local/bin
sudo cp example_config.yml /etc/nomnomlog-config.yml
sudo vi /etc/nomnomlog-config.ymlConfiguration directives can also be specified as command-line arguments (below).
Usage of nomnomlog 0.1.0:
-c, --configfile string Path to config (default "/etc/nomnomlog-config.yml")
--debug-log-cfg string The debug log file; overridden by -D/--no-detach
-d, --dest-host string Destination syslog hostname or IP
-p, --dest-port int Destination syslog port (default 514)
--eventmachine-tail No action, provided for backwards compatibility
-f, --facility string Facility (default "user")
-h, --help Display this help message
--hostname string Local hostname to send from (default "octothorpe")
--log string Set loggo config, like: --log="<root>=DEBUG" (default "<root>=INFO")
--new-file-check-interval int How often to check for new files (seconds) (default 10)
-D, --no-detach Do not daemonize and detach from the terminal; overrides --debug-log-cfg
--no-eventmachine-tail No action, provided for backwards compatibility
--pid-file string Location of the PID file
--poll Detect changes by polling instead of inotify
-s, --severity string Severity (default "notice")
--tcp Connect via TCP (no TLS)
--tls Connect via TCP with TLS
--truncate-hostname Local truncate-hostname to send from
-V, --version Display version and exitDaemonize and collect messages from files listed in ./example_config.yml as
well as the file /var/log/mysqld.log. Write PID to /tmp/nomnomlog.pid
and send to port logs.papertrailapp.com:12345:
$> nomnomlog -c example_config.yml -p 12345 --pid-file=/tmp/nomnomlog.pid /var/log/mysqld.log
Stay attached to the terminal, look for and use /etc/nomnomlog-config.yml if it
exists, and send with facility local0 to a.example.com:514:
$> nomnomlog -D -d a.example.com -f local0 /var/log/mysqld.log
nomnomlog will daemonize by default.
Sample init files can be found in the examples directory. You may be able to:
$> cp examples/nomnomlog.init.d /etc/init.d/nomnomlog
$> chmod 755 /etc/init.d/nomnomlog
And then ensure it's started at boot, either by using:
$> sudo update-rc.d nomnomlog defaults
or by creating a link manually:
$> sudo ln -s /etc/init.d/nomnomlog /etc/rc3.d/S30nomnomlog
With systemd it is also very simple with init.d defaults or drop in the systemd.service file :
admin@system1:/etc$ sudo service nomnomlog status
● nomnomlog.service - LSB: Start and Stop
Loaded: loaded (/etc/init.d/nomnomlog; generated)
Active: active (running) since Mon 2019-01-14 19:49:25 EST; 4s ago
Docs: man:systemd-sysv-generator(8)
Process: 17024 ExecStart=/etc/init.d/nomnomlog start (code=exited, status=0/SUCCESS)
Tasks: 9 (limit: 4663)
CGroup: /system.slice/nomnomlog.service
└─17037 nomnomlog -c /etc/nomnomlog-config.yml --pid-file=/var/run/nomnomlog.pid
Jan 14 19:49:25 system1 systemd[1]: Starting LSB: Start and Stop...
Jan 14 19:49:25 system1 nomnomlog[17024]: Starting nomnomlog
Jan 14 19:49:25 system1 systemd[1]: Started LSB: Start and Stop.
Additional information about init files (init.d, supervisor, systemd and upstart) are
available in the examples directory.
If the receiving system supports sending syslog over TCP with TLS, you can
pass the --tls option when running nomnomlog:
$> nomnomlog -D --tls -p 1234 /var/log/mysqld.log
or add protocol: tls to your configuration file.
By default, nomnomlog looks for a configuration in /etc/nomnomlog-config.yml.
The archive comes with a sample config. Optionally:
$> cp example_config.yml.example /etc/nomnomlog-config.yml
nomnomlog-config.yml has filenames to log from (as an array) and hostname and port
to log to (as a hash). Wildcards are supported using * and standard shell
globbing. Filenames given on the command line are additive to those in
the config file.
Only 1 destination server is supported; the command-line argument wins.
files:
- /var/log/httpd/access_log
- /var/log/httpd/error_log
- /var/log/mysqld.log
- /var/run/mysqld/mysqld-slow.log
destination:
host: logs.papertrailapp.com
port: 12345
protocol: tlsnomnomlog sends the name of the file without a path ("mysqld.log") as the syslog tag (program name).
After changing the configuration file, restart nomnomlog using the
init script or by manually killing and restarting the process. For example:
$> /etc/init.d/nomnomlog restart
Here's an advanced config which uses all options.
Provide --hostname somehostname or use the hostname configuration option:
hostname: somehostnamenomnomlog automatically detects and activates new log files that match
its file specifiers. For example, *.log may be provided as a file specifier,
and nomnomlog will detect a some.log file created after it was started.
By default, globs are re-checked every 10 seconds. To check for new files more
frequently, use the --new-file-check-interval argument. For example, to
recheck globs every 1 second, use:
--new-file-check-interval 1
Note: messages may be written to new files in the period between when the file is created and when the periodic glob check detects it. This data is not transmitted.
If globs are specified on the command-line, enclose each one in single-quotes
('*.log') so the shell passes the raw glob string to nomnomlog (rather
than the current set of matches). This is not necessary for globs defined in
the config file.
External log rotation scripts often move or remove an existing log file
and replace it with a new one (at a new inode). The Linux standard script
logrotate supports a copytruncate config
option. With that option, logrotate will copy files, operate on the copies,
and truncate the original so that the inode remains the same.
nomnomlog will handle both approaches seamlessly, so it should be no
concern as to which method is used. If a log file is moved or renamed,
and a new file is created (at a new inode), nomnomlog will follow that
new file at the new inode (assuming it has the same absolute path name). If
a file is copied then truncated, nomnomlog will seek to the beginning of
the truncated file and continue to read it.
Some logging programs such as Java's gclog (-XX:+PrintGC or -verbose:gc)
do not log in append mode, so if another program such as logrotate (set to
copytruncate) truncates the file, on the next write of the Java logger, the
OS will fill the file with NUL bytes up to the current offset of the file descriptor.
More info on that here.
nomnomlog will detect those leading NUL bytes, discard them, and log the discard count.
Provide one or more regular expressions to prevent certain files from being matched.
exclude_files:
- \.\d$
- .bz2
- .gznomnomlog includes logic to have both inclusion and exclusion patterns for syslog forwarding.
Filter order to collect and send line data is:
If line matches ((include_patterns and does not match exclude patterns) or does not match exclude_patterns) then send line.
There may be certain times when you only want certain log messages to be sent. These may be Alert level log lines that are mixed into a TRACE level log. You can filter out all other lines easily from the respective application. To filter these lines, use the include_patterns with an array or regexes:
include_patterns:
- include this
- alertThere may be certain log messages that you do not want to be sent. These may be repetitive log lines that are "noise" that you might not be able to filter out easily from the respective application. To filter these lines, use the exclude_patterns with an array or regexes:
exclude_patterns:
- exclude this
- \d+ thingsRun multiple instances to specify unique syslog hostnames.
To do that, provide an alternate PID path as a command-line option to the additional instance(s). For example:
--pid-file=/var/run/nomnomlog_2.pid
Note: Daemonized programs use PID files to identify whether the program is already running (more). Like other daemons, nomnomlog will refuse to run as a daemon (the default mode) when a PID file is present. If a .pid file is present but the daemon is not actually running, remove the PID file.
nomnomlog uses the log file name (like "access_log") as the syslog program name, or what the syslog RFCs call the "tag." This is ideal unless nomnomlog watches many files that have the same name.
In that case, tell nomnomlog to set another program name using the
tag attribute in the configuration file:
files:
- path: /var/log/httpd/access_log
tag: apache
destination:
host: logs.papertrailapp.com
port: 12345
protocol: tls... or on the command line:
$> nomnomlog apache=/var/log/httpd/access_logThis functionality was introduced in version 0.17
Linux signals can be caught and handled by the daemon. As an example SIGHUP will dump the current running configuration.
2018-11-20 19:59:58 INFO nomnomlog.go:50 Connecting to 127.0.0.1:5514 over udp
2018-11-20 19:59:58 INFO nomnomlog.go:197 Forwarding file: locallog.txt
Handling signal: hangup
Running Configuration:
{
"ExcludeFiles": [
"\\.DS_Store"
],
"ExcludePatterns": [
"don't log on me",
"do \\w+ on me"
],
"IncludePatterns": [
"log only me",
"log o.{1,2} me"
],
"ConnectTimeout": 5000000000,
"KeepReconnecting": true,
"WriteTimeout": 30000000000,
"NewFileCheckInterval": 10000000000,
"LogLevels": "\u003croot\u003e=INFO",
"DebugLogFile": "/dev/null",
"PidFile": "/var/run/nomnomlog.pid",
"TcpMaxLineLength": 99991,
"NoDetach": true,
"TCP": false,
"TLS": false,
"TruncateHostname": false,
"Files": [
{
"Path": "locallog.txt",
"Tag": ""
},
{
"Path": "/var/log/**/*.log",
"Tag": ""
},
{
"Path": "/var/log/nginx/nginx.log",
"Tag": "nginx"
},
{
"Path": "/var/log/httpd/access_log",
"Tag": "apache"
}
],
"Hostname": "octothorpe",
"Severity": 5,
"Facility": 1,
"Poll": false,
"Destination": {
"Host": "logs.papertrailapp.com",
"Port": 514,
"Protocol": "tls"
},
"RootCAs": {}
}To output debugging events with maximum verbosity, run:
$> nomnomlog --debug-log-cfg=logfile.txt --log="<root>=DEBUG".. as well as any other arguments which are used in normal operation. This
will set loggo's
root logger to the DEBUG level and output to logfile.txt.
To manage it on the command line use:
./build/nomnomlog/nomnomlog -D -c /etc/nomnomlog-config.yml --log="<root>=Trace"
2018-12-17 17:55:31 INFO nomnomlog.go:50 Connecting to 127.0.0.1:5514 over udp
2018-12-17 17:55:31 DEBUG nomnomlog.go:183 Evaluating globs every 10s
2018-12-17 17:55:31 DEBUG nomnomlog.go:199 Evaluating file globs
2018-12-17 17:55:31 INFO nomnomlog.go:218 Forwarding file: /var/log/syslog
2018-12-17 17:55:31 TRACE worker_registry.go:40 Adding /var/log/syslog to worker registry
2018-12-17 17:55:41 DEBUG nomnomlog.go:199 Evaluating file globsTo send messages longer than 1024 characters, use TCP (either TLS or cleartext TCP) instead of UDP. See "Sending messages securely" to use TCP with TLS for messages of any length.
Here's why longer UDP messages are impossible to send over the Internet.
When running nomnomlog in the foreground using the -D switch, if you
receive the error:
Error creating fsnotify watcher: inotify_init: too many open files
determine the maximum number of inotify instances that can be created using:
$> cat /proc/sys/fs/inotify/max_user_instances
and then increase this limit using:
$> echo VALUE >> /proc/sys/fs/inotify/max_user_instances
where VALUE is greater than the present setting. Confirm that nomnomlog starts
up and then apply this new value permanently by adding the following to
/etc/sysctl.conf::
fs.inotify.max_user_instances = VALUE
When monitoring a large number of files, this error may occur:
FATAL -- Error watching /path/here : no space left on device
To solve this, determine the maximum number of user watches that can be created using:
$> cat /proc/sys/fs/inotify/max_user_watches
and then increase them using:
$> echo VALUE >> /proc/sys/fs/inotify/max_user_watches
Once again, confirm that nomnomlog starts and then apply this value permanently by adding the following to /etc/sysctl.conf::
fs.inotify.max_user_watches = VALUE
A local release build which is golang based is very easy to build and does not require many external dependencies.
A distro-release is wrapped in a rpm or deb requires additional tooling that may not be necessary for individual usage.
The current convention is to use genus.species of ants for release names (Tetramorium.insolens).
nomnomlog is written in go, and uses govendor to manage dependencies. To get everything set up, install go then run:
$> go get github.com/kardianos/govendor
$> go get github.com/mitchellh/gox
$> go get github.com/shadowbq/nomnomlogDistro Release building requirements are checked in the Makefile
$> go get github.com/giantswarm/semver-bump
Distro builds may also include:
- ruby gem fpm (gem install -y fpm)
- rpmbuild (apt install -y rpm)
- etc..
OSX for OSX build.
$> make
Product Version 0.1.0
Checking Dependencies ---->
[..sic..]
fpm is not installed. See https://github.com/jordansissel/fpmChange the version in the VERSION file:
$> semver-bump minor-release
Build a local-release
$> ./build.sh
Building nomnomlog 0.1.0 local-release, use Makefile for distro-releases.
$> ./build/nomnomlog/nomnomlog --version
nomnomlog 0.1.0Build the distro-release
$> make
$> make test
To run tests manually:
# run all tests
$> go test ./...
# run all tests except the slower syslog reconnection tests
$> go test -short ./...Travis-ci.org will push to github.com our successful builds when tagged with "v*"
From the CLI, add, commit, push to master. Then tag (annotate), and push the tag. This will cause a release cycle for travis to push the build files (deb,rpm,etc.) to repo release pages.
$> git add .
$> git commit -m 'updated to v1.0'
[master ba27ddc] updated to v1.0
1 file changed, 1 insertion(+), 1 deletion(-)
$> git push origin master
...
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/shadowbq/nomnomlog
f747451..ba27ddc master -> master
$> git tag -a 'v1.0' -m 'major version release'
$> git push origin tag v1.0
...
Counting objects: 9, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 939 bytes | 234.00 KiB/s, done.
Total 9 (delta 6), reused 0 (delta 0)
remote: Resolving deltas: 100% (6/6), completed with 4 local objects.
To https://github.com/shadowbq/nomnomlog
* [new tag] v1.0 -> v1.0At the bottom of the Travis log you should see
Preparing deploy
Logged in as shadowbq
Deploying to repo: shadowbq/nomnomlog
Current tag is: v1.0
Deploying application
Setting target_commitish to ba27ddc24d03a7e706f7af4a32a0b0773fdc018e
Done. Your build exited with 0.
$> make
build
clean
depend
package
reportVersion
tarball
test
darwin/
linux/
windows/Current ARM builds support all ARM platforms with hardware floating point instruction sets. This includes All Raspberry PI devices, most ARMv6 chips (Cortex), and ARMv7 and beyond.
Once you've made your great commits:
- Fork nomnomlog
- Create a topic branch -
git checkout -b my_branch - Commit the changes without changing the Rakefile or other files unrelated to your enhancement.
- Push to your branch -
git push origin my_branch - Create a Pull Request or an Issue with a link to your branch
- That's it!
- Github Contributions
- remote_syslog2 original fork
- See whether the issue has already been reported: https://github.com/shadowbq/nomnomlog/issues/
- If you don't find one, create an issue with a repro case.
