Pipeline Collisions Occur When They Shouldn't
epociask opened this issue · comments
Bug Description
Current DAG implementation will add edges between existing components based on conflicting ComponentID
values. However, this means that pipelines that perform backfills can conflict with ones that are live. Same for backtesting pipelines as well.
Example Scenario
Two pipelines are shown below (RP0, RP1)
, both of which use the same exact components, but require backfilling from some starting heights (x0, x1)
where:
x in Z+ and x <= current_block_height
In this example, since RP0.component_set = RP1.component_set
, RP1
will be treated as a duplicate pipeline with the EtlManager since their respective pipeline IDs are equal. This will result in RP1
powering some invariant from whatever point in chain history that RP0 is in; resulting in invariants failing to successfully backfill.
It's important to note that in this scenario, x0
could equal x1
but doesn't necessarily have to. The presence of some x
for a registerPipeline denotes that it has backfill
requirements to be ran. This means that each pipeline state is:
I. Syncing when monotonic x
< current_block_height
II. Live once x
>= current_block_height
Problem Solution
Introduce access management logic that performs additional integrity checks before reusing an existing component. I,e. The presence of sync
flag for a pipeline deems it globally unique where it's components cannot be reused.
In the example of identical pipelines (P0, P1) with the same IDs but backfilling toggles, the pipelines would merge once their states have transitioned (syncing --> live
).