[BUG] dbt deps permission denied for dbt-expectations
christopherekfeldt opened this issue · comments
Is this a new bug in dbt-expectations?
- I believe this is a new bug in dbt-expectations
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
When running poetry run dbt deps it fails during installing of the packages on dbt-expectations giving error messages regarding permissions denied.
Expected Behavior
To dbt deps to run successfully and install all packages specified in the packages.yml
Steps To Reproduce
Our packages.yml config:
packages:
-
package: dbt-labs/dbt_utils
version: 1.0.0 -
package: dbt-labs/codegen
version: 0.9.0 -
package: elementary-data/elementary
version: 0.7.1 -
package: calogica/dbt_expectations
version: 0.8.2
run dbt deps or poetry run dbt deps
Relevant log output
08:40:51 Installed from version 0.7.1
08:40:51 Updated version available: 0.7.3
08:40:51 Installing calogica/dbt_expectations
08:40:56 Encountered an error:
[Errno 13] Permission denied: 'integration_tests'
08:40:56 Traceback (most recent call last):
File "/usr/lib/python3.8/shutil.py", line 791, in move
os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: 'dbt_packages/dbt-expectations-0.8.2' -> 'dbt_packages/dbt_expectations'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 136, in main
results, succeeded = handle_and_check(args)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 206, in handle_and_check
task, res = run_from_args(parsed)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 253, in run_from_args
results = task.run()
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/task/deps.py", line 75, in run
package.install(self.config, renderer)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/registry.py", line 60, in install
self._install(project, renderer)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/base.py", line 117, in _install
connection_exception_retry(download_untar_fn, 5)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/utils.py", line 616, in _connection_exception_retry
return fn()
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/base.py", line 129, in download_and_untar
system.untar_package(tar_path, deps_path, package_name)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/clients/system.py", line 497, in untar_package
dbt.clients.system.rename(downloaded_path, desired_path, force=True)
File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/clients/system.py", line 485, in rename
shutil.move(from_path, to_path)
File "/usr/lib/python3.8/shutil.py", line 809, in move
rmtree(src)
File "/usr/lib/python3.8/shutil.py", line 718, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.8/shutil.py", line 659, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 657, in _rmtree_safe_fd
os.rmdir(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'integration_tests'
Environment
- OS: Windows leveraging WSL
- Python: 3.8.10
- dbt: 1.4.4
- dbt-expectations: 0.8.2
Which database adapter are you using with dbt?
dbt-bigquery 1.4.1
Note: dbt-expectations currently does not support database adapters other than the ones listed below.
- Postgres
- Snowflake
- BigQuery
Additional Context
We have an ugly solution now that works. Steps to make it work:
- Remove dbt-expectations from packages.yml
- Run dbt deps
- Successfull installation
- Add dbt-expectations 0.8.2 to packages.yml
- Run dbt deps again
- Succesfull installation
Hi, thanks for opening this issue! I can't repro this locally, so just a couple of comments:
- this may or may not be related, but I don't recommend ever pinning to point releases (e.g. 0.8.2), but instead I recommend pinning to major release ranges for all packages, not just dbt-expectations, e.g.
packages:
- package: calogica/dbt_expectations
version: [">=0.8.0", "<0.9.0"]
The default instructions on dbt-hub are misleading there imo. That way you get automatic bug fixes etc.
- if you're getting permissions errors during
dbt deps
, I think that's more likely an issue with dbt-core, or something else locally, since dbt-expectations is just a git repo you're cloning duringdbt deps
. - curious, what if you remove the elementary package, do you still get the error? They have some integrations with dbt-expectations and I wonder if that's causing it. They're on 0.7.5 btw.
Thanks @clausherther So that range you put out there, how is it translated exactly?
Does it say take some version of dbt_expectations between 0.8.0 and 0.9.0 which is compatible? Or does it always select the latest? Or does it try with the latest, and if that doesn't work it tries with an older version?
Using the range ensures you're automatically getting the latest minor release for each package, and that you don't end up with weird incompatibilities between packages.