GoogleCloudPlatform / mlops-on-gcp

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

lab-02-kfp-pipeline: kpf BQ component run failure

BenoitDherin opened this issue · comments

When running the pipeline from the notebook two BQ components are failing:

image

It seems that the first component that succeeds creates a dataset, and the two other components that are failing are doing so because that dataset has already been created by that first component:

image

The pipeline succeeds though when run directly from the UI.

This is a race condition between multiple parallel components. I thought about fixing it but I decided not to as this is in fact a good pedagogical tool to talk about proper pipeline design and to show how to retry the pipeline. The pipeline will succeed after at most two attempts. You will find the comment in the notebook that refers to this error scenario.