logfellow / logstash-logback-encoder

Logback JSON encoder and appenders

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Enhancement request] Publish logs to Elastic DataStream

patpatpat123 opened this issue · comments

Hello team,

I would like to reach out with a small enhancement request, please.

Currently, this logstash logback encoder is doing a great job forwarding application logs to Logstash (which will then send them to ElasticSearch).
However, in ElasticSearch, they live inside a regular index.

Would it be possible from this project to send the logs to ElasticSearch, but to a DataStream, not just a regular index:
https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html

(Copy paste from ElasticSearch official doc:)
A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data.

(see screenshot for datastream)
Screenshot 2023-04-11 163729

Currently, this is only create a plain index (please see screenshot)

Screenshot 2023-04-11 163814

Rationale
The reason behind this request is that by definition, logs are very well suited for ElasticSearch DataStream, because they are data stream in the first place.

However, as of now, one needs to manually convert the log index into DataStream from the regular Index on the ElasticSearch backend in order to benefit from the synergy between this encoder and elasticsearch data stream.

Would be a great feature if this encoder can directly push the logs as it is doing today, directly to a data stream instead.

Thank you