dbt-labs / dbt-event-logging

a dbt package to make auditing dbt runs easy.

Home Page:https://hub.getdbt.com/dbt-labs/logging/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Log select/exclude arguments with invocations

falonso-alo opened this issue · comments

Describe the feature

It would be useful to log run select/exclude arguments. For example, if the following commands were run:

# run a single model
dbt run --select mymodel

# run a single tag
dbt run --select tag:mytag

# run a single tag with an exclude
dbt run --select tag:mytag --exclude no_not_that_model

Then it would be nice if the select and exclude arguments were logged as well. For example:

invocation_id event_name select exclude
1 run started mymodel null
2 run started tag:mytag null
3 run started tag:mytag no_not_that_model

Of course, the columns should probably not be SQL keywords, but the exact column name wording is probably out of scope the request.

Describe alternatives you've considered

It might be possible to use a pre-hook to fetch that information an insert record in a table somewhere in the database. However, there are some downsides:

  1. This would be cumbersome
  2. It's unclear if this should go into the existing dbt_meta.dbt_audit_log table
  3. This fits thematically with the logging package

Additional context

When there are potentially multiple DBT jobs running, it can be difficult to pin point which job is related to which invocation. Logging the select/exclude arguments would make it easier to tie the logging to a particular job.

Who will this benefit?

People who have multiple DBT runs every day that target different tags/models.