calogica / dbt-expectations

Port(ish) of Great Expectations to dbt test macros

Home Page:https://calogica.github.io/dbt-expectations/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] dbt deps permission denied for dbt-expectations

christopherekfeldt opened this issue · comments

Is this a new bug in dbt-expectations?

  • I believe this is a new bug in dbt-expectations
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When running poetry run dbt deps it fails during installing of the packages on dbt-expectations giving error messages regarding permissions denied.

Expected Behavior

To dbt deps to run successfully and install all packages specified in the packages.yml

Steps To Reproduce

Our packages.yml config:

packages:

  • package: dbt-labs/dbt_utils
    version: 1.0.0

  • package: dbt-labs/codegen
    version: 0.9.0

  • package: elementary-data/elementary
    version: 0.7.1

  • package: calogica/dbt_expectations
    version: 0.8.2

run dbt deps or poetry run dbt deps

Relevant log output

08:40:51 Installed from version 0.7.1
08:40:51 Updated version available: 0.7.3
08:40:51 Installing calogica/dbt_expectations
08:40:56 Encountered an error:
[Errno 13] Permission denied: 'integration_tests'
08:40:56 Traceback (most recent call last):
File "/usr/lib/python3.8/shutil.py", line 791, in move
os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: 'dbt_packages/dbt-expectations-0.8.2' -> 'dbt_packages/dbt_expectations'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 136, in main
results, succeeded = handle_and_check(args)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 206, in handle_and_check
task, res = run_from_args(parsed)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 253, in run_from_args
results = task.run()
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/task/deps.py", line 75, in run
package.install(self.config, renderer)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/registry.py", line 60, in install
self._install(project, renderer)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/base.py", line 117, in _install
connection_exception_retry(download_untar_fn, 5)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/utils.py", line 616, in _connection_exception_retry
return fn()
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/base.py", line 129, in download_and_untar
system.untar_package(tar_path, deps_path, package_name)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/clients/system.py", line 497, in untar_package
dbt.clients.system.rename(downloaded_path, desired_path, force=True)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/clients/system.py", line 485, in rename
shutil.move(from_path, to_path)
File "/usr/lib/python3.8/shutil.py", line 809, in move
rmtree(src)
File "/usr/lib/python3.8/shutil.py", line 718, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.8/shutil.py", line 659, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 657, in _rmtree_safe_fd
os.rmdir(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'integration_tests'

Environment

- OS: Windows leveraging WSL
- Python: 3.8.10
- dbt: 1.4.4
- dbt-expectations: 0.8.2

Which database adapter are you using with dbt?

dbt-bigquery 1.4.1
Note: dbt-expectations currently does not support database adapters other than the ones listed below.

  • Postgres
  • Snowflake
  • BigQuery

Additional Context

We have an ugly solution now that works. Steps to make it work:

  1. Remove dbt-expectations from packages.yml
  2. Run dbt deps
  3. Successfull installation
  4. Add dbt-expectations 0.8.2 to packages.yml
  5. Run dbt deps again
  6. Succesfull installation

Hi, thanks for opening this issue! I can't repro this locally, so just a couple of comments:

  • this may or may not be related, but I don't recommend ever pinning to point releases (e.g. 0.8.2), but instead I recommend pinning to major release ranges for all packages, not just dbt-expectations, e.g.
packages:
  - package: calogica/dbt_expectations
    version: [">=0.8.0", "<0.9.0"]

The default instructions on dbt-hub are misleading there imo. That way you get automatic bug fixes etc.

  • if you're getting permissions errors during dbt deps, I think that's more likely an issue with dbt-core, or something else locally, since dbt-expectations is just a git repo you're cloning during dbt deps.
  • curious, what if you remove the elementary package, do you still get the error? They have some integrations with dbt-expectations and I wonder if that's causing it. They're on 0.7.5 btw.

Thanks @clausherther So that range you put out there, how is it translated exactly?

Does it say take some version of dbt_expectations between 0.8.0 and 0.9.0 which is compatible? Or does it always select the latest? Or does it try with the latest, and if that doesn't work it tries with an older version?

Using the range ensures you're automatically getting the latest minor release for each package, and that you don't end up with weird incompatibilities between packages.