Giters

ArroyoSystems / arroyo

Distributed stream processing engine in Rust

Home Page:https://arroyo.dev

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Better support previewing multiple queries

mwylde opened this issue · comments

commented

Currently the preview experience works like this:

  • We plan the query
  • We compile it into a dataflow
  • We traverse the dataflow and find all of the sinks and replace them with WebSinks

Then on the frontend, we request the /v1/pipelines/{pipeline_id}/jobs/{job_id}/output API which makes a request to the controller to stream the job outputs.

This then merges all of the outputs into a single result table, which can be pretty hard to parse.

Instead we should separate out the outputs for each query (possibly into separate tabs in the UI) and allow distinguishing which operator they came from the in API.