Go-Fluentd

Rewrite fluentd-server by Golang, Higher performance with less resource requirement.

At-Least-Once guarantee(disk WAL)
log concatenation by head regexp expression
log parsing by regexp expression(support embedded json)
log filter by custom plugins(acceptorfilters & tagfilters)
multiple receivers(support multiple protocols: msgpack, http, syslog, kafka, ...)
multiple senders(support multiple backend: elasticsearch, fluentd, ...)
multiple environments deployment(--env: sit, perf, uat, prod)

Already running on our PRODUCION since 2018/9.

When processing 1000mbps logs flood:

dstat
monitor
profile

Documents:

Description

LogAggregator + Concator + Parser + Producer.

Origin logs emitted from docker look like:

'{"container_id": "xxxxx", "log": "2018-03-06 16:56:22.514 | mscparea | ERROR  | http-nio-8080-exec-1 | com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest | 84:'
'{"container_id": "xxxxx", "log": "Exception in thread "main" java.lang.IllegalStateException: A book has a null property"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Author.getBookIds(Author.java:38)"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)"}'
'{"container_id": "xxxxx", "log": "Caused by: java.lang.NullPointerException"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Book.getId(Book.java:22)"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Author.getBookIds(Author.java:35)"}'
'{"container_id": "xxxxx", "log": "\t... 1 more"}'

After Concator(TagPipeline > concator_f):

&FluentMsg{
    Id: 12345,
    Tag: "spring.sit",
    Message: map[string]interface{}{
        "container_id": "xxxxx",
        "log": "2018-03-06 16:56:22.514 | mscparea | ERROR  | http-nio-8080-exec-1 | com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest | 84: Exception in thread "main" java.lang.IllegalStateException: A book has a null property\n\tat com.example.myproject.Author.getBookIds(Author.java:38)\n\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\nCaused by: java.lang.NullPointerException\n\tat com.example.myproject.Book.getId(Book.java:22)\n\tat com.example.myproject.Author.getBookIds(Author.java:35)\n\t... 1 more",
    },
}

After Parser(TagPipeline > parser_f):

&FluentMsg{
    Id: 12345,
    Tag: "spring.sit",
    Message: map[string]interface{}{
        "container_id": "xxxxx",
        "time": "2018-03-06 16:56:22.514",
        "level": "ERROR",
        "app": "mscparea",
        "thread": "http-nio-8080-exec-1",
        "class": "com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest",
        "line": 84,
        "message": "Exception in thread "main" java.lang.IllegalStateException: A book has a null property\n\tat com.example.myproject.Author.getBookIds(Author.java:38)\n\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\nCaused by: java.lang.NullPointerException\n\tat com.example.myproject.Book.getId(Book.java:22)\n\tat com.example.myproject.Author.getBookIds(Author.java:35)\n\t... 1 more",
    },
}

Then Producer can send logs to anywhere (depends on Senders).

Run

directly run:

go run -race main.go \
  --config=./docs/settings/tiny_settings.yml \
  --env=sit \
  --log-level=debug

run by docker:

docker run -itd --rm --name=go-fluentd -p 24225:24225 -p 8080:8080 \
    -v /etc/configs/go-fluentd:/etc/go-fluentd \
    -v /data/log/fluentd/go-fluentd:/data/log/fluentd/go-fluentd
    ppcelery/go-fluentd:1.13.1 \
    ./go-fluentd \
        --config=/etc/go-fluentd/settings.yml \
        --env=perf \
        --addr=0.0.0.0:8080
        --host=x.x.x.x
        --enable-auto-gc

docker images version

stable
release
dev
<feature taskid>

Roles

Acceptor (consists of Recvs)
AcceptorPipeline (consists of AcceptorFilters)
Journal
Dispatcher
TagPipeline (consists of TagFilters)
- Concator
- Parser for each tag
PostPipeline (consists of PostFilters)
Producer (consists of Senders)

Acceptor

Contains multiply Recvs (such as KafkRecv & FluentdRecv), can listening tcp port or fetch msg from kafka brokers.

AcceptorPipeline

Contains multiply AcceptorFilters, be used for ignore or retag specific messages. All filters should return very fast to avoid blocking.

Journal

...

Dispatcher

...

TagPipeline

...

PostPipeline

...

Producer

...

Laisky / go-fluentd