Giters

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

Home Page:https://clear.ml/docs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Use pipeline artifacts as input to step functions

jmw52 opened this issue · comments

commented

Proposal Summary

It would be great to be able to add an artifact to the pipeline, and use it as input to a function step, something like:

pipe = PipelineController(name="my_pipeline", project="my_project")
pipe.upload_artifact(name="my_artifact", artifact_object=my_artifact)
pipe.add_function_step(
    name="step",
    function=step_function,
    function_kwargs=dict(param="${pipeline.artifacts.my_artifact}"),
)

Currently it is only possible to do this with parameters not artifacts, which is limiting.

Motivation

For pipelines with multiple function steps which take as input the same object. Currently multiple artifacts are created from the same object e.g. step_1.my_artifact and step_2.my_artifact. Using a single pipeline.my_artifact would solve this duplication.

Related Discussion

Slack thread: https://clearml.slack.com/archives/CTK20V944/p1706861297432839