[BUG] IndexError: list index out of range

[Sunny Shah]

Describe the bug
This bug is coming from dbterd package.

To Reproduce
Steps to reproduce the behavior:

1. Install dbterd
2. dbt docs gererate
3. dbterd run -ad /path/to/target/ -o "/path/to/output

Expected behavior
I expect it to run successfully.

Screenshots
If applicable, add screenshots to help explain your problem.

$ dbterd run -ad path/to/target/ -o path/to/output
2024-01-01 15:23:06,665 - dbterd - INFO - Run with dbterd==1.7.2 (main.py:54)
2024-01-01 15:23:06,666 - dbterd - INFO - Using dbt project dir at: /Users/sunnyshah/CloudFactory/edw-dbt/target (base.py:45)
2024-01-01 15:23:06,666 - dbterd - INFO - Using dbt artifact dir at: /Users/sunnyshah/CloudFactory/edw-dbt/target (base.py:69)
Traceback (most recent call last):
  File "/Users/sunnyshah/.pyenv/versions/edw_dbt/bin/dbterd", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/main.py", line 6, in main
    cli.dbterd()
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/cli/params.py", line 117, in wrapper
    return func(*args, **kwargs)  # pragma: no cover
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/cli/main.py", line 75, in run
    Executor(ctx).run(**kwargs)
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/base.py", line 30, in run
    self.__run_by_strategy(**kwargs)
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/base.py", line 162, in __run_by_strategy
    result = operation(manifest=manifest, catalog=catalog, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/targets/dbml/dbml_test_relationship.py", line 16, in run
    return ("output.dbml", parse(manifest, catalog, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/targets/dbml/dbml_test_relationship.py", line 29, in parse
    tables, relationships = test_relationship.parse(
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 38, in parse
    relationships = get_relationships(manifest=manifest, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 75, in get_relationships
    refs = [
           ^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 78, in <listcomp>
    table_map=get_table_map(test_node=manifest.nodes[x], **kwargs),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 174, in get_table_map
    if f'("{map[1].split(".")[-1]}")'.lower() in to_model.replace("'", '"').lower():
            ~~~^^^
IndexError: list index out of range

Desktop (please complete the following information):

• OS: Macbook Pro Sonoma 14.2.1 (23C71)
• dbterd version: 1.7.2

Additional context
Add any other context about the problem here.

[Dat Nguyen]

Hi @Sunnyinho Thanks for finding this issue, could you help to give me more info in order to reproduce the issue please?

• dbt --version info
• If possible, can you help to send me the files: manifest.json and catalog.json which are located under the target dir? You could send it to my email ([e-mail]) if it is confidential

Thank you again

[Sunny Shah]

Hi @Sunnyinho Thanks for finding this issue, could you help to give me more info in order to reproduce the issue please?

• dbt --version info
• If possible, can you help to send me the files: manifest.json and catalog.json which are located under the target dir? You could send it to my email ([e-mail]) if it is confidential

Thank you again

dbt core: 1.6.0
dbt snowflake: 1.6.1

I've sent a mail for manifest.json and catalog.json

[Dat Nguyen]

Hi @Sunnyinho thanks for sending the files. I had a try on them and confirmed that it was failing ⚠️

Technically, in your manifest.json file, the relationship test node doesn't have and depends_on nodes information -- this is really unusual behavior in dbt. I need some more time to check it out further 🏃

In the meantime, would you like to try upgrading your dbt 1.6 to the latest and try again for any lucks? 👀

• dbt core 1.6 latest is 1.6.9
• dbt snowflake 1.6 latest is 1.6.6
  Actually, I have the jaffle_shop locally with 1.6.6 working perfectly. Please help to let me know whether it could resolve the issue.

Thanks

[Sunny Shah]

Hi @Sunnyinho thanks for sending the files. I had a try on them and confirmed that it was failing ⚠️

Technically, in your manifest.json file, the relationship test node doesn't have and depends_on nodes information -- this is really unusual behavior in dbt. I need some more time to check it out further 🏃

In the meantime, would you like to try upgrading your dbt 1.6 to the latest and try again for any lucks? 👀

• dbt core 1.6 latest is 1.6.9
• dbt snowflake 1.6 latest is 1.6.6
  Actually, I have the jaffle_shop locally with 1.6.6 working perfectly. Please help to let me know whether it could resolve the issue.

Thanks

Hi @datnguye thanks for checking out. I also tried with dbt-core version 1.7.4 and dbt-snowflake 1.7.1 but the issue still persists.

[Dat Nguyen]

Hi @Sunnyinho Thanks for the info.

Just had a deep dive looking into manifest.json file, and found the related model (model.edw.agt_datasets) of the relationship test is disabled which explained why we had no data in depends_on nodes.

Can you try again with using the Select or Exclude option to exclude all disabled models?

Thanks! Let me know if it would be working after that
PS: Sorry, this is a limitation in dbt itself, but I will keen to hear your idea to bypass this issue in dbterd

[Sunny Shah]

Thanks for the support @datnguye. Going to close the ticket now