allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

Home Page:https://clear.ml/docs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Better info about pipeline step status

materight opened this issue · comments

Proposal Summary

Show the current progress and correct elapsed time of running steps in the pipeline view.

Motivation

Currently the running time always shows >1s instead of the actual elapsed time.
image

It also doesn't show the current progress, it would be nice to have a progress bar similar to the one shown in the experiment list
image

Thanks for the feedback @materight.

Indeed there seems to be an issue in presenting elapsed time - We'll work on a fix for a near version.

For progress indication - Do note that steps progress needs to be explicitly specified by your step implementation (see Task.set_progress())

Hi @ainoam, thanks for the update.

Re progress indication: in this example I'm calling set_progress() on the step task, but the progress bar is shown only if I open the task list view, while I'm proposing to show it also in the pipeline view.

Thanks for clarifying @materight.

Makes total sense - We'll look into supporting this in a near version.

Hey @materight! Just letting you know that the issue of the pipeline step duration not updating has been resolved in the recently released clearml-server 1.13.0. Let us know if there are any issues :)

@pollfly thanks for the update. Unfortunately it doesn't seem to be working... Now it doesn't show anything, even after the step has been completed.
image

@materight Thank you for spotting. We'll take a look and will update when the issue is smoothed out :)