Giters

dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Home Page:https://getdbt.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Feature] progress report just like argo workflow does

matepek opened this issue · comments

commented

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Hello,
I'm using argo workflow to run dbt and it works fine.
Recently I have realised that argo workflow has a feature to report and show the progress of the step.

It's fairly simple. If any process writes the coming from environment variable ARGO_PROGRESS_FILE with format like 2/10 then argo will display it. (See argo documentation for details). One thing worth to mention, it seems that the file needs a new line character at the end which is not documented, so the actual value should be 2/10\n.

So if dbt could have the same option with their own variable like DBT_PROGRESS_FILE then I could just export ARGO_PROGRESS_FILE=$DBT_PROGRESS_FILE and I'm done.

Argo workflow UI:
Screenshot 2024-04-25 at 12 01 14

Describe alternatives you've considered

I have checked callbacks but it des not tell anything about the number of tasks dbt run/build will process. But it could be special message coming from the callback which tells this number. I could parse that and process it but it's quite cumbersome and hard to maintain.

Who will this benefit?

Yeah, it's not a dealbreaker but also sound like quick improvement. I can contribute too..

Are you interested in contributing this feature?

it might just 5 min for maintainers but I can contribute too

Anything else?

Thanks for your consideraton.

commented

Thanks for reaching out @matepek !

dbt-core already provides a generalized events logging that can be used to translate into other formats (like that defined by ARGO_PROGRESS_FILE). See also: logs.

This provides the general capabilities without dbt-core being responsible to provide translations to other formats.

This specific feature request for a DBT_PROGRESS_FILE is out of scope for dbt-core, so I'm closing this as "not planned".

commented

Cool. I found the index and total fields. Seems it will work for me. Thanks