stg_dbt_audit_log relation reference does not work when using custom `generate_schema_name` macro
clrcrl opened this issue · comments
Claire Carroll commented
Steps to reproduce
- Add a custom
generate_schema_name
macro to your project, e.g.:
$ cat macros/generate_schema_name.sql
{% macro generate_schema_name(custom_schema_name, node) -%}
{%- set default_schema = target.schema -%}
{%- if custom_schema_name is none -%}
{{ default_schema }}_foo
{%- else -%}
{{ custom_schema_name | trim }}_foo
{%- endif -%}
{%- endmacro %}
- Install this package, and add post-hooks to project.yml
$ cat packages.yml
packages:
- local: "/Users/claire/fishtown/packages/logging"
$ cat dbt_project.yml
name: 'jaffle_shop'
version: '0.1'
profile: 'jaffle_shop'
source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]
target-path: "target"
clean-targets:
- "target"
- "dbt_modules"
- "logs"
models:
jaffle_shop:
materialized: table
staging:
materialized: view
pre-hook: "{{ logging.log_model_start_event() }}"
post-hook: "{{ logging.log_model_end_event() }}"
- dbt run
Expected results
Success
Actual results
Error - the dbt_audit_log
SQL is invalid
$ dbt run
Running with dbt=0.15.0
Found 11 models, 20 tests, 0 snapshots, 0 analyses, 264 macros, 4 operations, 3 seed files, 0 sources
11:11:33 |
11:11:33 | Running 3 on-run-start hooks
11:11:33 | 1 of 3 START hook: logging.on-run-start.0............................ [RUN]
11:11:33 | 1 of 3 OK hook: logging.on-run-start.0............................... [CREATE SCHEMA in 0.06s]
11:11:33 | 2 of 3 START hook: logging.on-run-start.1............................ [RUN]
11:11:33 | 2 of 3 OK hook: logging.on-run-start.1............................... [CREATE TABLE in 0.05s]
11:11:33 | 3 of 3 START hook: logging.on-run-start.2............................ [RUN]
11:11:33 | 3 of 3 OK hook: logging.on-run-start.2............................... [INSERT 0 1 in 0.10s]
11:11:33 |
11:11:33 | Concurrency: 1 threads (target='dev_redshift')
11:11:33 |
11:11:33 | 1 of 11 START view model jaffle_shop_dev_foo.stg_orders.............. [RUN]
11:11:35 | 1 of 11 OK created view model jaffle_shop_dev_foo.stg_orders......... [CREATE VIEW in 1.42s]
11:11:35 | 2 of 11 START view model jaffle_shop_dev_foo.stg_payments............ [RUN]
11:11:36 | 2 of 11 OK created view model jaffle_shop_dev_foo.stg_payments....... [CREATE VIEW in 1.36s]
11:11:36 | 3 of 11 START view model meta_foo.stg_dbt_audit_log.................. [RUN]
11:11:37 | 3 of 11 ERROR creating view model meta_foo.stg_dbt_audit_log......... [ERROR in 0.89s]
11:11:37 | 4 of 11 START view model jaffle_shop_dev_foo.stg_customers........... [RUN]
11:11:39 | 4 of 11 OK created view model jaffle_shop_dev_foo.stg_customers...... [CREATE VIEW in 1.53s]
11:11:39 | 5 of 11 START table model jaffle_shop_dev_foo.customer_orders........ [RUN]
11:11:41 | 5 of 11 OK created table model jaffle_shop_dev_foo.customer_orders... [SELECT in 1.51s]
11:11:41 | 6 of 11 START table model jaffle_shop_dev_foo.customer_payments...... [RUN]
11:11:43 | 6 of 11 OK created table model jaffle_shop_dev_foo.customer_payments. [SELECT in 1.76s]
11:11:43 | 7 of 11 START table model jaffle_shop_dev_foo.order_payments......... [RUN]
11:11:45 | 7 of 11 OK created table model jaffle_shop_dev_foo.order_payments.... [SELECT in 1.81s]
11:11:45 | 8 of 11 SKIP relation meta_foo.stg_dbt_deployments................... [SKIP]
11:11:45 | 9 of 11 SKIP relation meta_foo.stg_dbt_model_deployments............. [SKIP]
11:11:45 | 10 of 11 START table model jaffle_shop_dev_foo.fct_customers......... [RUN]
11:11:47 | 10 of 11 OK created table model jaffle_shop_dev_foo.fct_customers.... [SELECT in 2.54s]
11:11:47 | 11 of 11 START table model jaffle_shop_dev_foo.fct_orders............ [RUN]
11:11:50 | 11 of 11 OK created table model jaffle_shop_dev_foo.fct_orders....... [SELECT in 1.95s]
11:11:50 |
11:11:50 | Running 1 on-run-end hook
11:11:50 | 1 of 1 START hook: logging.on-run-end.0.............................. [RUN]
11:11:50 | 1 of 1 OK hook: logging.on-run-end.0................................. [COMMIT in 0.30s]
11:11:50 |
11:11:50 |
11:11:50 | Finished running 6 view models, 5 table models, 4 hooks in 17.96s.
Completed with 1 error and 0 warnings:
Database Error in model stg_dbt_audit_log (models/stg_dbt_audit_log.sql)
relation "meta_foo.dbt_audit_log" does not exist
compiled SQL at target/run/logging/stg_dbt_audit_log.sql
The problematic SQL is:
with audit as (
select * from {{this.schema}}.dbt_audit_log
),
...
Related to #7