[Pre-regression] Error when serializing RunResults (variable results) with to_dict() during the on-run-end
DmytroSly opened this issue · comments
Is this a new bug in dbt-core?
- I believe this is a new bug in dbt-core
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
I log RunResults into a database table by parsing the results variable this way:
{% macro parse_dbt_results(results) %}
-- Create a list of parsed results
{%- set parsed_results = [] %}
-- Flatten results and add to list
{% for run_result in results %}
-- Convert the run result object to a simple dictionary
{% set run_result_dict = run_result.to_dict() %}
{% do parsed_results.append(run_result_dict) %}
{% endfor %}
{{ return(parsed_results) }}
{% endmacro %}
After upgrading to dbt-core==1.8.0rc1 and dbt-snowflake==1.8.0b3 I get the following error:
Encountered an error:
Class RunResult has unresolved type reference agate in some of its fields
09:15:00 Traceback (most recent call last):
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 138, in wrapper
result, success = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 101, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 218, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 247, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 294, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 332, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\main.py", line 568, in run
results = task.run()
^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 526, in run
result = self.execute_with_hooks(selected_uids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 488, in execute_with_hooks
self.after_run(adapter, res)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 484, in after_run
self.safe_run_hooks(adapter, RunHookType.End, extras)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 423, in safe_run_hooks
self.run_hooks(adapter, hook_type, extra_context)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 374, in run_hooks
sql = self.get_hook_sql(adapter, hook, idx, num_hooks, extra_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 326, in get_hook_sql
compiled = self.compiler.compile_node(hook, self.manifest, extra_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 533, in compile_node
node = self._compile_code(node, manifest, extra_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 421, in _compile_code
node.compiled_code = jinja.get_rendered(
^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - SIBASE_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 146, in get_rendered
rendered = render_template(template, ctx, node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 539, in render_template
return template.render(ctx)
^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 144, in render
return self.environment.handle_exception()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\environment.py", line 939, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
return self.call_macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
return macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
rv = self._func(*arguments)
^^^^^^^^^^^^^^^^^^^^^^
File "<template>", line 8, in template
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
return self.call_macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
return macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
rv = self._func(*arguments)
^^^^^^^^^^^^^^^^^^^^^^
File "<template>", line 9, in template
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 2, in __mashumaro_to_dict__
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 1140, in add_pack_method
self._add_pack_method_lines(method_name)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 836, in _add_pack_method_lines
field_types = self.get_field_types(include_extras=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 221, in get_field_types
return self.__get_field_types(include_extras=include_extras)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 188, in __get_field_types
raise UnresolvedTypeReferenceError(self.cls, name) from None
mashumaro.exceptions.UnresolvedTypeReferenceError: Class RunResult has unresolved type reference agate in some of its fields
Expected Behavior
No error happens, just like in the previous versions of dbt-core
Steps To Reproduce
Just try to serialize the results variable during the on-run-end in the below way:
{% macro parse_dbt_results(results) %}
-- Create a list of parsed results
{%- set parsed_results = [] %}
-- Flatten results and add to list
{% for run_result in results %}
-- Convert the run result object to a simple dictionary
{% set run_result_dict = run_result.to_dict() %} -- the error happens here
{% do parsed_results.append(run_result_dict) %}
{% endfor %}
{{ return(parsed_results) }}
{% endmacro %}
Relevant log output
Encountered an error:
Class RunResult has unresolved type reference agate in some of its fields
09:15:00 Traceback (most recent call last):
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 138, in wrapper
result, success = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 101, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 218, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 247, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 294, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 332, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\main.py", line 568, in run
results = task.run()
^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 526, in run
result = self.execute_with_hooks(selected_uids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 488, in execute_with_hooks
self.after_run(adapter, res)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 484, in after_run
self.safe_run_hooks(adapter, RunHookType.End, extras)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 423, in safe_run_hooks
self.run_hooks(adapter, hook_type, extra_context)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 374, in run_hooks
sql = self.get_hook_sql(adapter, hook, idx, num_hooks, extra_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 326, in get_hook_sql
compiled = self.compiler.compile_node(hook, self.manifest, extra_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 533, in compile_node
node = self._compile_code(node, manifest, extra_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 421, in _compile_code
node.compiled_code = jinja.get_rendered(
^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - SIBASE_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 146, in get_rendered
rendered = render_template(template, ctx, node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 539, in render_template
return template.render(ctx)
^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 144, in render
return self.environment.handle_exception()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\environment.py", line 939, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
return self.call_macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
return macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
rv = self._func(*arguments)
^^^^^^^^^^^^^^^^^^^^^^
File "<template>", line 8, in template
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
return self.call_macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
return macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
rv = self._func(*arguments)
^^^^^^^^^^^^^^^^^^^^^^
File "<template>", line 9, in template
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 2, in __mashumaro_to_dict__
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 1140, in add_pack_method
self._add_pack_method_lines(method_name)
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 836, in _add_pack_method_lines
field_types = self.get_field_types(include_extras=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 221, in get_field_types
return self.__get_field_types(include_extras=include_extras)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 188, in __get_field_types
raise UnresolvedTypeReferenceError(self.cls, name) from None
mashumaro.exceptions.UnresolvedTypeReferenceError: Class RunResult has unresolved type reference agate in some of its fields
Environment
- OS: Windows 10 Enterprise
- Python: 3.11.4
- dbt-core: 1.8.0rc1
Which database adapter are you using with dbt?
dbt-snowflake==1.8.0b3
Additional Context
No response
Thanks for reporting this @DmytroSly !
I was able to see the same difference as you between 1.7 and 1.8 (see "reprex" below for details). I didn't confirm or deny, but it might be related to the changes in #9744.
Reprex
Add the following to dbt_project.yml
:
on-run-end: "{{ log_dbt_results(results) }}"
models/my_model.sql
select 1 as id
macros/log_dbt_results.sql
{% macro parse_dbt_results(results) %}
-- Create a list of parsed results
{%- set parsed_results = [] %}
{{ log("results: " ~ results, True) }}
-- Flatten results and add to list
{% for run_result in results %}
-- Convert the run result object to a simple dictionary
{% set run_result_dict = run_result.to_dict() %}
{% do parsed_results.append(run_result_dict) %}
{% endfor %}
{{ return(parsed_results) }}
{% endmacro %}
{% macro log_dbt_results(results) %}
{% set parsed_results = parse_dbt_results(results) %}
{% for run_result in results %}
-- Convert the run result object to a simple dictionary
{{ log("run_result: " ~ run_result, True) }}
{% endfor %}
-- dummy SQL to execute in the on-run-end hook
select 2 as id
{% endmacro %}
Thanks for the report @DmytroSly, and for the quick follow-up @dbeatty10!
I can confirm that updating this line to remove the conditional if TYPE_CHECKING
makes the reproduction case pass. I suspect adding that back in would undo the startup performance improvement, but it might be our safest path toward avoiding this regression before the v1.8.0 final release.
For context, here is where we're adding results: List[RunResult]
into the on-run-end
Jinja context:
Line 470 in fb6dbc8
Alternative ideas:
- Exclude
agate_table
from theto_dict
serialization ofRunResult
. (We haven't documented that it's included, and I believe it's only populated for tests, seeds, anddbt show
.) - Change the type of what's added into the
on-run-end
context fromList[RunResult]
(which includesnode
andagate_table
) toList[RunResultOutput]
(which doesn't, and exactly matches what's inrun_results.json
). This would be a breaking change, since we've documented thatnode
is included: https://docs.getdbt.com/reference/dbt-classes#result-objects
@jtcohen6 It looks like agate_table is already excluded, in the sense that it always explicitly serializes to None, so it's unfortunate that mashumaro gets so upset that its annotated type isn't available.
There's a variant of your second suggestion that might work. Instead of RunResult, we could provide instances of its base class NodeResult to on-run-end, which does include node, but not agate_table.
@peterallenwebb and I agreed that:
- The "undo" is fastest & safest fix for v1.8.0-final
- It would be great to redo this speedup, possibly by converting
RunResult -> NodeResult
before sticking in theon-run-end
context (Peter's suggestion) - We need to add a test case for this: "on-run-end: [
{{ results[0].to_dict() }}
] does not raise anUnresolvedTypeReferenceError
"