Giters

PrefectHQ / prefect-dbt

Collection of Prefect integrations for working with dbt with your Prefect flows.

Home Page:https://prefecthq.github.io/prefect-dbt/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

PermissionError: [Errno 13] Permission denied: 'dbt_logs'

opened this issue · comments

commented

I am getiing the following error and I am not sure how to resolve.

Code:

from prefect import flow
from prefect_dbt.cli.commands import trigger_dbt_cli_command
import os

@flow
def trigger_dbt_cli_command_flow() -> str:
    os.chdir('../dbtcode/MDP')
    result = trigger_dbt_cli_command("dbt run --select example")
    return result # Returns the last line the in CLI output

trigger_dbt_cli_command_flow()

Error:


12:15:13.828 | INFO    | prefect.engine - Created flow run 'lavender-skunk' for flow 'trigger-dbt-cli-command-flow'
12:15:14.010 | INFO    | Flow run 'lavender-skunk' - Created task run 'trigger_dbt_cli_command-321ca940-0' for task 'trigger_dbt_cli_command'
12:15:14.011 | INFO    | Flow run 'lavender-skunk' - Executing 'trigger_dbt_cli_command-321ca940-0' immediately...
12:15:14.046 | INFO    | Task run 'trigger_dbt_cli_command-321ca940-0' - Running dbt command: dbt run --select example --profiles-dir /.dbt
12:15:16.386 | INFO    | Task run 'trigger_dbt_cli_command-321ca940-0' - 12:15:16  Encountered an error:
[Errno 13] Permission denied: 'dbt_logs'

12:15:16.503 | ERROR   | Task run 'trigger_dbt_cli_command-321ca940-0' - Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1062, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cli/commands.py", line 158, in trigger_dbt_cli_command
    result = await shell_run_command.fn(command=command, **shell_run_command_kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_shell/commands.py", line 90, in shell_run_command
    raise RuntimeError(msg)
RuntimeError: Command failed with exit code 2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 129, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 225, in run_from_args
    log_manager.set_path(log_path)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 514, in set_path
    self._file_handler.set_path(path)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 405, in set_path
    make_log_dir_if_missing(log_dir)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 340, in make_log_dir_if_missing
    dbt.clients.system.make_directory(log_dir)
  File "/usr/local/lib/python3.10/site-packages/dbt/clients/system.py", line 112, in make_directory
    raise e
  File "/usr/local/lib/python3.10/site-packages/dbt/clients/system.py", line 106, in make_directory
    os.makedirs(path)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: 'dbt_logs'

12:15:16.538 | ERROR   | Task run 'trigger_dbt_cli_command-321ca940-0' - Finished in state Failed('Task run encountered an exception.')
12:15:16.538 | ERROR   | Flow run 'lavender-skunk' - Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 566, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/prefectcode/dbt_example.py", line 8, in trigger_dbt_cli_command_flow
    result = trigger_dbt_cli_command("dbt run --select example")
  File "/usr/local/lib/python3.10/site-packages/prefect/tasks.py", line 294, in __call__
    return enter_task_run_engine(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 705, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 136, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 822, in create_task_run_then_submit
    return await future._result()
  File "/usr/local/lib/python3.10/site-packages/prefect/futures.py", line 227, in _result
    return final_state.result(raise_on_failure=raise_on_failure)
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1062, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cli/commands.py", line 158, in trigger_dbt_cli_command
    result = await shell_run_command.fn(command=command, **shell_run_command_kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_shell/commands.py", line 90, in shell_run_command
    raise RuntimeError(msg)
RuntimeError: Command failed with exit code 2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 129, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 225, in run_from_args
    log_manager.set_path(log_path)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 514, in set_path
    self._file_handler.set_path(path)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 405, in set_path
    make_log_dir_if_missing(log_dir)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 340, in make_log_dir_if_missing
    dbt.clients.system.make_directory(log_dir)
  File "/usr/local/lib/python3.10/site-packages/dbt/clients/system.py", line 112, in make_directory
    raise e
  File "/usr/local/lib/python3.10/site-packages/dbt/clients/system.py", line 106, in make_directory
    os.makedirs(path)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: 'dbt_logs'

12:15:16.573 | ERROR   | Flow run 'lavender-skunk' - Finished in state Failed('Flow run encountered an exception.')
Traceback (most recent call last):
  File "/prefectcode/dbt_example.py", line 11, in <module>
    trigger_dbt_cli_command_flow()
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 384, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 152, in enter_flow_run_engine_from_flow_call
    return anyio.run(begin_run)
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 105, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 228, in create_then_begin_flow_run
    return state.result()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 566, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/prefectcode/dbt_example.py", line 8, in trigger_dbt_cli_command_flow
    result = trigger_dbt_cli_command("dbt run --select example")
  File "/usr/local/lib/python3.10/site-packages/prefect/tasks.py", line 294, in __call__
    return enter_task_run_engine(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 705, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 136, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 822, in create_task_run_then_submit
    return await future._result()
  File "/usr/local/lib/python3.10/site-packages/prefect/futures.py", line 227, in _result
    return final_state.result(raise_on_failure=raise_on_failure)
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1062, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cli/commands.py", line 158, in trigger_dbt_cli_command
    result = await shell_run_command.fn(command=command, **shell_run_command_kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_shell/commands.py", line 90, in shell_run_command
    raise RuntimeError(msg)
RuntimeError: Command failed with exit code 2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 129, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.10/site-packages/dbt/main.py", line 225, in run_from_args
    log_manager.set_path(log_path)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 514, in set_path
    self._file_handler.set_path(path)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 405, in set_path
    make_log_dir_if_missing(log_dir)
  File "/usr/local/lib/python3.10/site-packages/dbt/logger.py", line 340, in make_log_dir_if_missing
    dbt.clients.system.make_directory(log_dir)
  File "/usr/local/lib/python3.10/site-packages/dbt/clients/system.py", line 112, in make_directory
    raise e
  File "/usr/local/lib/python3.10/site-packages/dbt/clients/system.py", line 106, in make_directory
    os.makedirs(path)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: 'dbt_logs'
commented

What if you do not os.chdir('../dbtcode/MDP')? It seems like dbt does not have permissions to create the directory dbt_logs in there. Can you try running the code snippet directly in ../dbtcode/MDP?

commented

Hi @ahuang11 , you are right. I resolved this issue by running the command export DBT_LOG_PATH = /dbt_logs and chmod 755 -R /dbt_logs to the dbt user.

commented

Awesome to hear, and thanks for sharing!