dbt-labs / dbt-event-logging

a dbt package to make auditing dbt runs easy.

Home Page:https://hub.getdbt.com/dbt-labs/logging/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

stg_dbt_audit_log relation reference does not work when using custom `generate_schema_name` macro

clrcrl opened this issue · comments

Steps to reproduce

  1. Add a custom generate_schema_name macro to your project, e.g.:
$ cat macros/generate_schema_name.sql
{% macro generate_schema_name(custom_schema_name, node) -%}
    {%- set default_schema = target.schema -%}
    {%- if custom_schema_name is none -%}
        {{ default_schema }}_foo
    {%- else -%}
        {{ custom_schema_name | trim }}_foo
    {%- endif -%}
{%- endmacro %}
  1. Install this package, and add post-hooks to project.yml
 $ cat packages.yml
packages:
  - local: "/Users/claire/fishtown/packages/logging"
 $ cat dbt_project.yml

name: 'jaffle_shop'
version: '0.1'
profile: 'jaffle_shop'
source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]

target-path: "target"
clean-targets:
    - "target"
    - "dbt_modules"
    - "logs"

models:
  jaffle_shop:
      materialized: table
      staging:
        materialized: view
  pre-hook: "{{ logging.log_model_start_event() }}"
  post-hook: "{{ logging.log_model_end_event() }}"
  1. dbt run

Expected results

Success

Actual results

Error - the dbt_audit_log SQL is invalid

$ dbt run
Running with dbt=0.15.0
Found 11 models, 20 tests, 0 snapshots, 0 analyses, 264 macros, 4 operations, 3 seed files, 0 sources

11:11:33 |
11:11:33 | Running 3 on-run-start hooks
11:11:33 | 1 of 3 START hook: logging.on-run-start.0............................ [RUN]
11:11:33 | 1 of 3 OK hook: logging.on-run-start.0............................... [CREATE SCHEMA in 0.06s]
11:11:33 | 2 of 3 START hook: logging.on-run-start.1............................ [RUN]
11:11:33 | 2 of 3 OK hook: logging.on-run-start.1............................... [CREATE TABLE in 0.05s]
11:11:33 | 3 of 3 START hook: logging.on-run-start.2............................ [RUN]
11:11:33 | 3 of 3 OK hook: logging.on-run-start.2............................... [INSERT 0 1 in 0.10s]
11:11:33 |
11:11:33 | Concurrency: 1 threads (target='dev_redshift')
11:11:33 |
11:11:33 | 1 of 11 START view model jaffle_shop_dev_foo.stg_orders.............. [RUN]
11:11:35 | 1 of 11 OK created view model jaffle_shop_dev_foo.stg_orders......... [CREATE VIEW in 1.42s]
11:11:35 | 2 of 11 START view model jaffle_shop_dev_foo.stg_payments............ [RUN]
11:11:36 | 2 of 11 OK created view model jaffle_shop_dev_foo.stg_payments....... [CREATE VIEW in 1.36s]
11:11:36 | 3 of 11 START view model meta_foo.stg_dbt_audit_log.................. [RUN]
11:11:37 | 3 of 11 ERROR creating view model meta_foo.stg_dbt_audit_log......... [ERROR in 0.89s]
11:11:37 | 4 of 11 START view model jaffle_shop_dev_foo.stg_customers........... [RUN]
11:11:39 | 4 of 11 OK created view model jaffle_shop_dev_foo.stg_customers...... [CREATE VIEW in 1.53s]
11:11:39 | 5 of 11 START table model jaffle_shop_dev_foo.customer_orders........ [RUN]
11:11:41 | 5 of 11 OK created table model jaffle_shop_dev_foo.customer_orders... [SELECT in 1.51s]
11:11:41 | 6 of 11 START table model jaffle_shop_dev_foo.customer_payments...... [RUN]
11:11:43 | 6 of 11 OK created table model jaffle_shop_dev_foo.customer_payments. [SELECT in 1.76s]
11:11:43 | 7 of 11 START table model jaffle_shop_dev_foo.order_payments......... [RUN]
11:11:45 | 7 of 11 OK created table model jaffle_shop_dev_foo.order_payments.... [SELECT in 1.81s]
11:11:45 | 8 of 11 SKIP relation meta_foo.stg_dbt_deployments................... [SKIP]
11:11:45 | 9 of 11 SKIP relation meta_foo.stg_dbt_model_deployments............. [SKIP]
11:11:45 | 10 of 11 START table model jaffle_shop_dev_foo.fct_customers......... [RUN]
11:11:47 | 10 of 11 OK created table model jaffle_shop_dev_foo.fct_customers.... [SELECT in 2.54s]
11:11:47 | 11 of 11 START table model jaffle_shop_dev_foo.fct_orders............ [RUN]
11:11:50 | 11 of 11 OK created table model jaffle_shop_dev_foo.fct_orders....... [SELECT in 1.95s]
11:11:50 |
11:11:50 | Running 1 on-run-end hook
11:11:50 | 1 of 1 START hook: logging.on-run-end.0.............................. [RUN]
11:11:50 | 1 of 1 OK hook: logging.on-run-end.0................................. [COMMIT in 0.30s]
11:11:50 |
11:11:50 |
11:11:50 | Finished running 6 view models, 5 table models, 4 hooks in 17.96s.

Completed with 1 error and 0 warnings:

Database Error in model stg_dbt_audit_log (models/stg_dbt_audit_log.sql)
  relation "meta_foo.dbt_audit_log" does not exist
  compiled SQL at target/run/logging/stg_dbt_audit_log.sql

The problematic SQL is:

with audit as (

    select * from {{this.schema}}.dbt_audit_log

),
...

Related to #7