base-org / pessimism

Detect real-time threats and events on OP Stack compatible blockchains

Home Page:https://base-org.github.io/pessimism/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Pipeline Collisions Occur When They Shouldn't

epociask opened this issue · comments

commented

Bug Description

Current DAG implementation will add edges between existing components based on conflicting ComponentID values. However, this means that pipelines that perform backfills can conflict with ones that are live. Same for backtesting pipelines as well.

Example Scenario

Two pipelines are shown below (RP0, RP1), both of which use the same exact components, but require backfilling from some starting heights (x0, x1) where:

x in Z+ and x <= current_block_height

Pessimism - Merging Pipelines (2)

In this example, since RP0.component_set = RP1.component_set, RP1 will be treated as a duplicate pipeline with the EtlManager since their respective pipeline IDs are equal. This will result in RP1 powering some invariant from whatever point in chain history that RP0 is in; resulting in invariants failing to successfully backfill.

It's important to note that in this scenario, x0 could equal x1 but doesn't necessarily have to. The presence of some x for a registerPipeline denotes that it has backfill requirements to be ran. This means that each pipeline state is:
I. Syncing when monotonic x < current_block_height
II. Live once x >= current_block_height

Problem Solution

Introduce access management logic that performs additional integrity checks before reusing an existing component. I,e. The presence of sync flag for a pipeline deems it globally unique where it's components cannot be reused.

In the example of identical pipelines (P0, P1) with the same IDs but backfilling toggles, the pipelines would merge once their states have transitioned (syncing --> live).