allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

Home Page:https://clear.ml/docs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Updating pipeline without re-running task

LifeBoey opened this issue · comments

Hi there,

I'd like to ask here the same question I did in slack (https://clearml.slack.com/archives/C028LNAAR5H/p1695283353097409 ), which is regarding updating the pipeline and running the updated pipeline, without having to re-run the affected tasks.

If I were to change the code in the pipeline, I find that simply running the pipeline again would use the code from the previously ran instance, even after updating the code in the git repository. In order for the pipeline to acknowledge the new code, it'd require me to run the individual task(s) that were affected by it.

I was wondering if there is just a faster way to update the pipeline code without having to go through the tedium of running the tasks again. Much appreciated!

Hi @LifeBoey ! Just for the record, you can override repo info in add_step using task_overrides https://clear.ml/docs/latest/docs/references/sdk/automation_controller_pipelinecontroller#add_step