Giters

deis / logger

In-memory log buffer used by Deis Workflow.

Home Page:https://deis.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Proposal] Move from syslog endpoints to using straight tcp

jchauncey opened this issue · comments

commented

Recently we have noticed issues with duplicate log messages within the logger and stdout metrics components. These duplicate messages are only from components running on the same host as logger. While I am unsure of the root cause of this it has pointed out a deficiency on how we handle log messages within the Workflow stack. My proposal is as follows:

Proposal

Create a deis specific fluentd plugin that understands the log message structure and forwards that data directly to logger. This means instead of sending several thousand log messages a second to logger we only send the data we care about storing within logger itself. We can even massage the data to make it easier for logger to find the information it needs more quickly.

This plugin can also do the same thing for sending data to stdout metrics and only send the router log stream. We could even explore having the plugin send data directly to influx which eliminates the stdout metrics component completely.

Pluses

  • Moves logic on filter data closer to the source instead of having it duplicated in two different components
  • Reduces the number of components we need to run
  • Eliminate the duplicate log message problem we are seeing
  • Easy to extend

Minuses

  • We need to build and release a fluentd plugin which is shipped as a ruby gem
  • Ruby vs Go
commented

ping @jchauncey on possibility of moving this to a later milestone