dbt-labs / dbt-event-logging

a dbt package to make auditing dbt runs easy.

Home Page:https://hub.getdbt.com/dbt-labs/logging/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Allow writing audit logs to a custom schema

Limess opened this issue · comments

Describe the feature

Currently audit logs are written to the output of the get_audit_schema macro (https://github.com/fishtown-analytics/dbt-event-logging/blob/c0232661597039ae266d63836faf59d262fb5043/macros/audit.sql#L9). This is explicitly namespaced using logging in all places of use so I believe it is impossible to override this in the consuming project.

Now that the target is logged in the audit logs, it would be useful to be able to override the audit log schema so as to be able to log to the same table and report based on target, e.g. we run test and local builds with a different schema and different targets but we can't customise our BI tool to vary the schema dynamically so can't do this in a single report.

I believe the implementation would require removing the logging. namespacing from this macro, allowing users to add a macro with the same name to their projects which is used instead?

Describe alternatives you've considered

Using a DBT macro to union tables across different schemas based on a LIKE pattern.

Who will this benefit?

Anyone who wants to visualise DBT performance across different target schemas using a single audit log table.

I'm all for this! dbt gets a little weird when calling a non-namespaced macro from a package (I think it defaults to the package version of the macro, but I'm happy to be proven wrong). If you can hold on for 0.16.0, you can make the logic reasonably explicit:

    {# if the get_audit_schema macro exists in the base project #}
    {% if context.get(project_name, {}).get('get_audit_schema')  %}
        {% set audit_schema=context[project_name].get_audit_schema(**kwargs) %}
    {% else %}
        {% set audit_schema=logging.get_audit_schema(**kwargs) %}
    {% endif %}

^ There might be further tidying you can do here.

Looks great! For our use case we can wait for 0.16.0, this primarily for future tracking build times on CI which we haven't yet implemented.