abecciu / fluentd

Fluentd data collector, Log Everything in JSON

Fluentd¶ ↑

Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages.

Architecture¶ ↑

Fluentd collects events from various data sources and writes them to files, database or other types of storages:

Web apps  ---+                 +--> file
             |                 |
             +-->           ---+
/var/log  ------>  Fluentd  ------> mail
             +-->           ---+
             |                 |
Apache    ----                 +--> Fluentd

Fluent also supports log transfer:

Web server
+---------+
| Fluentd -------
+---------+|     |
 +---------+     |
                 |
Proxy server     |    Log server, Amazon S3, HDFS, ...
+---------+      +--> +---------+
| Fluentd ----------> | Fluentd ||
+---------+|     +--> +---------+|
 +---------+     |     +---------+
                 |
Database server  |
+---------+      |
| Fluentd ---------> mail
+---------+|
 +---------+

An event consists of tag, time and record. Tag is a string separated with ‘.’ (e.g. myapp.access). It is used to categorize events. Time is a UNIX time recorded at occurrence of an event. Record is a JSON object.

Quick Start¶ ↑

$ gem install fluentd
$ # install sample configuration file to the directory
$ fluentd -s conf
$ fluentd -c conf/fluent.conf &
$ echo '{"json":"message"}' | fluent-cat debug.test

Meta¶ ↑

Web site

fluentd.org/

Documents

docs.fluentd.org/

Source repository

github.com/fluent

Discussion

groups.google.com/group/fluentd

Author

Sadayuki Furuhashi

Copyright

© 2011 FURUHASHI Sadayuki

License

Apache License, Version 2.0

Contributors:¶ ↑

Patches contributed by those people.