Stream Pipelines

[Omkar Kesarkhane]

Which OpenObserve functionalities are relevant/related to the feature request?

streams

Description

Overview:
Stream pipelines offer a visual and intuitive way to process streams. With an easy drag-and-drop interface, users can construct stream pipelines, add conditional routing which creates new streams from existing stream, and associate functions with ease. This feature enhances the platform's capability to handle streaming data efficiently.

Proposed solution

Features:

1. Stream Pipeline Creation:
   Users can easily create stream pipelines using a drag-and-drop interface.
2. Conditional Routing (Stream Creation):
   Conditional routing of streams allows users to create new streams based on predefined conditions from the existing stream.
3. Function Association:
   Users can associate functions with streams to process data as required.
4. Pipeline Visualization:
   Users can visualize the the pipeline, making it easy to understand and debug.

Flow of Pipeline
During execution of pipeline, routings will get executed before functions

Alternatives considered

no

[Omkar Kesarkhane]

Feature PR - #3282

[Juan Calderon-Perez]

@omkarK06 That's a pretty cool feature, are the functions a specific language? Having VRL there would be nice / performant.

[Neha P]

testing for this ticket is complete