[Bug] dbt_project.yml not found when dbt_deps=True in RenderConfig and dependencies are pre-installed

[Ryan Hatter]

Astronomer Cosmos Version

Other Astronomer Cosmos version (please specify below)

If "Other Astronomer Cosmos version" selected, which one?

1.4.3

dbt-core version

1.7.16

Versions of dbt adapters

dbt-sqlserver==1.7.4

LoadMode

AUTOMATIC

ExecutionMode

LOCAL

InvocationMode

None

airflow version

2.9.1

Operating System

Debian GNU/Linux 11 (bullseye)

If a you think it's an UI issue, what browsers are you seeing the problem on?

No response

Deployment

Astronomer

Deployment details

No response

What happened?

Tasks intermittently fail with the below traceback

Relevant log output

[2024-06-14, 04:55:51 UTC] {logging_mixin.py:188} INFO - 04:55:51 Encountered an error:
[Errno 2] No such file or directory: '/opt/dbt/packages/codegen/dbt_project.yml'
[2024-06-14, 04:55:51 UTC] {logging_mixin.py:188} INFO - 04:55:51 Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/dbt/cli/requires.py", line 91, in wrapper
result, success = func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/dbt/cli/requires.py", line 76, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/dbt/cli/requires.py", line 169, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/dbt/cli/requires.py", line 198, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/dbt/cli/requires.py", line 245, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/dbt/cli/requires.py", line 271, in wrapper
ctx.obj["manifest"] = parse_manifest(
File "/usr/local/lib/python3.10/site-packages/dbt/parser/manifest.py", line 1790, in parse_manifest
manifest = ManifestLoader.get_full_manifest(
File "/usr/local/lib/python3.10/site-packages/dbt/parser/manifest.py", line 311, in get_full_manifest
loader = cls(
File "/usr/local/lib/python3.10/site-packages/dbt/parser/manifest.py", line 262, in __init__
self.manifest.state_check = self.build_manifest_state_check()
File "/usr/local/lib/python3.10/site-packages/dbt/parser/manifest.py", line 978, in build_manifest_state_check
with open(path) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/dbt/packages/codegen/dbt_project.yml'

### How to reproduce

Set dbt_deps to True in RenderConfig (default) and pre-install dependencies when building the Astro image

### Anything else :)?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Contact Details

_No response_

[dosubot]

Hey there, @RNHTTR! 👋 I'm here to help you with any bugs, questions, or contributions you have. I'm an expert bot and can assist you while waiting for a human maintainer.

To resolve the intermittent FileNotFoundError for /opt/dbt/packages/codegen/dbt_project.yml in your Astronomer deployment, ensure that the necessary symlinks are created correctly and that the dbt_project.yml file exists in the expected location.

The create_symlinks function is responsible for creating symlinks to the dbt project files. Ensure that this function is called correctly and that the dbt_project.yml file is not in the ignore_paths list.

Here is the relevant part of the code that creates symlinks:

def create_symlinks(project_path: Path, tmp_dir: Path, ignore_dbt_packages: bool) -> None:
    """Helper function to create symlinks to the dbt project files."""
    ignore_paths = [DBT_LOG_DIR_NAME, DBT_TARGET_DIR_NAME, "profiles.yml"]
    if ignore_dbt_packages:
        ignore_paths.append("dbt_packages")
    for child_name in os.listdir(project_path):
        if child_name not in ignore_paths:
            os.symlink(project_path / child_name, tmp_dir / child_name)

Make sure that the dbt_project.yml file is present in the project_path and not excluded by the ignore_paths list. If the file is missing or incorrectly linked, it could cause the FileNotFoundError.

Additionally, verify that the project_path and tmp_dir are correctly set and accessible in your deployment environment. If the paths are incorrect or the files are not present, it could lead to intermittent errors.

If the issue persists, you may need to add logging to track the creation of symlinks and ensure that the dbt_project.yml file is correctly linked every time ^[1].

There are also known issues related to the dbt_deps attribute in the RenderConfig class that could cause intermittent FileNotFoundError issues. Setting dbt_deps=False can lead to a CosmosLoadDbtException due to missing dbt_packages, which could manifest as a FileNotFoundError in certain contexts ^[2].

To continue talking to Dosu, mention @dosu.

[Tatiana Al-Chueyr]

@RNHTTR if the dependencies are pre-installed, then dbt_deps should be False, not true.
If this is true, Cosmos will attempt to run dbt deps in the scheduler, and this command requires the dbt_project.yml file to exist.

[Ryan Hatter]

Should Cosmos raise an exception that makes that obvious? Or at least should the docs highlight that?