Giters

PrefectHQ / prefect-dbt

Collection of Prefect integrations for working with dbt with your Prefect flows.

Home Page:https://prefecthq.github.io/prefect-dbt/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Write Unit/Integration Tests

sungchun12 opened this issue · comments

commented

@desertaxle can you let me know your timeline to finish overall development?

I'm asking, so I can dedicate the right time and energy to help review PRs and contribute where you need help!

commented

@sungchun12 I expect to finish the last two major development issues (#9 and #10) this week. I think next week will be a good time for review/QA to make sure that I've covered everything that you'd like to see in this collection.

commented

Thanks Alex! I'll create time blocks in my calendar for next week.

commented

@belasobral93 FYI so you can plan your time out next week for UX testing!

commented

@sungchun12 @belasobral93 #9 and #10 have been completed, so prefect-dbt should be ready for UX testing. Let me know if there's anything from me that you need to support UX testing. FYI, Prefect is having planning meetings Tuesday through Friday of this week, so my responses might be delayed.

commented

@belasobral93

Can you check off all the below? If there's anything funky, can you make a loom video to show why the UX is funky or is not working?

From there, Alex can make new PRs to fix the issues you find!

  • Verify I can plug and play an API token, account id, job id into python code with minimal python knowledge
  • Verify I can trigger a job async
  • Verify I can trigger a job synchronously with polling
  • Verify I can get any artifact such as sources.json, run_results.json, catalog.json, manifest.json, example_sql.sql
  • Verify I can create a dbt Cloud job using an arbitrary config with json
  • Verify the prefect function docstrings are plainly understood from a data analyst perspective
  • Verify I can send custom trigger options such a step override commands from a previous task dynamically
  • Verify I get a dbt Cloud job to succeed with nice hyperlinks and logs
  • Verify I get a dbt Cloud job to fail with helpful hyperlinks and logs
commented

Allocated time to test this thursday : )

commented

Checking in to see if there were any issues discovered during testing. We're back from our planning meetings so we should be able to quickly jump on any issues that are found.

commented

Hi Alex,

I am running into issues in the installation (see error below after running pip install). I’m meeting with Sung this afternoon to see if I can get unblocked. If not, I’ll reach back out

Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x1067a8250>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/pip/

commented

Hi there,

Thanks for the patience in getting feedback on testing. I was unable to get flows to show up locally in the UI, so the following is from the command line. I'm about halfway complete, will finish the rest today (July 20)

  • Verify I can plug and play an API token, account id, job id into python code with minimal python knowledge
    • Plug and play was simple enough - I would ask to do a quick review of all the function names and how they are referenced
      in examples and import statements. I happened to choose trigger_dbt_cloud_job_run () and used the examples that
      referenced it as trigger_job_run() so experienced an error.

[❌] Verify I can trigger a job async (multi-threaded)

  • I’m attempting to use DaskTaskRunner but am getting:
   ImportError: cannot import name 'DaskTaskRunner' from 
  'prefect.task_runners'

Code:

import time
 
from prefect import task, flow
from prefect_dbt.cloud import DbtCloudCredentials
from prefect_dbt.cloud.jobs import trigger_dbt_cloud_job_run
 
@task
def call_api(job_id):
   job = job_id
   credentials = DbtCloudCredentials(api_key="d78e619a99c8acca50d691c4a9477d505756fec2", account_id=43799)
   trigger_dbt_cloud_job_run(dbt_cloud_credentials=credentials, job = job)
 
from prefect.task_runners import DaskTaskRunner
 
@flow(task_runner=DaskTaskRunner())
def async_flow():
   call_api(64242)
   call_api(77213)

  • Verify I can trigger a job synchronously (sequentially, one after the other) with polling

    • Yes. Created two flows, the first one with a while loop, and first flow completed before second flow was triggered

  • Verify I can get any artifact such as sources.json, run_results.json, catalog.json, manifest.json, example_sql.sql

    • Yes all of the above
commented

Thanks for the feedback @belasobral93!

We recently moved the Dask task runner out of the core library and into a seperate collection. To run Prefect with Dask, you'll need to install prefect-dask and import the Dask task runner from there.

Let me know if you run into any other issues and I'll be happy to help!

commented

Thanks @desertaxle . In trying to execute jobs asynchronously, I'm coming across a runtime warning in using trigger_dbt_cloud_job_run_and_wait_for_completion(). I'm also not seeing any logs indicating the flow was triggered.

error:

/Users/belaloaner/Desktop/prefect-dbt-cloud/dbt_cloud_example.py:7: RuntimeWarning: coroutine 'create_then_begin_flow_run' was never awaited
trigger_dbt_cloud_job_run_and_wait_for_completion(
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

code:

import asyncio
from prefect_dbt.cloud import DbtCloudCredentials
from prefect_dbt.cloud.jobs import trigger_dbt_cloud_job_run_and_wait_for_completion


async def main():
            trigger_dbt_cloud_job_run_and_wait_for_completion(
                dbt_cloud_credentials=DbtCloudCredentials(
                    api_key="d78e619a99c8acca50d691c4a9477d505756fec2",
                    account_id=43799
                ),
                job_id=77213
            )


asyncio.run(main())
commented

  • Verify I can plug and play an API token, account id, job id into python code with minimal python knowledge

  • Verify the prefect function docstrings are plainly understood from a data analyst perspective

    • @desertaxle I can’t see docstrings in VSCode. Does it have to do with in the import statements I’m seeing text such as: “Import prefect cannot be resolved”? I'm on Python 3.10.4

  • Verify I can trigger a job synchronously with polling

  • Verify I can get any artifact such as sources.json, run_results.json, catalog.json, manifest.json, example_sql.sql

  • Verify I can send custom trigger options such a step override commands from a previous task dynamically

    • yes using TriggerJobOptions()

  • [❌] Verify I can trigger a job async

    • Attempted to do so but would see first task completed before second task started. Also used await asyncio.sleep but that resulted in each task only beginning once the prior task completed. Is this user error?

code:

@task
async def trigger_dbt_cloud_job_run(job_id):
    await asyncio.sleep(5)
    return


@flow(task_runner=DaskTaskRunner())
def async_flow():
    print("job 1 called")
    trigger_dbt_cloud_job_run(64242)
    print("job 2 called")
    trigger_dbt_cloud_job_run(77213)
    print("job 3 called")
    trigger_dbt_cloud_job_run(77213)
    print("job 4 called")
    trigger_dbt_cloud_job_run(77213)


if __name__ == "__main__":
    async_flow()

  • [❌] Verify I can create a dbt Cloud job using an arbitrary config with json

  • [❌] Verify I get a dbt Cloud job to succeed with nice hyperlinks and logs

    • Please see above issue comment

  • [❌] Verify I get a dbt Cloud job to fail with helpful hyperlinks and logs

    • Please see above issue comment
commented

@belasobral93 "Verify I can create a dbt Cloud job using an arbitrary config with json"

Can you verify that code like this can work to create a brand new dbt cloud job?

I took this from the docstring in this codebase: https://github.com/PrefectHQ/prefect-dbt/blob/main/prefect_dbt/cloud/utils.py#L77

from prefect import flow

from prefect_dbt.cloud import DbtCloudCredentials
from prefect_dbt.cloud.utils import call_dbt_cloud_administrative_api_endpoint

@flow
def create_job_flow():
    credentials = DbtCloudCredentials(api_key="my_api_key", account_id=123456789)

    future = call_dbt_cloud_administrative_api_endpoint(
        dbt_cloud_credentials=credentials,
        path="/jobs/",
        http_method="POST",
        json={
            "account_id": 123456789,
            "project_id": 100,
            "environment_id": 10,
            "name": "Nightly run",
            "dbt_version": "0.17.1",
            "triggers": {"github_webhook": True, "schedule": True},
            "execute_steps": ["dbt run", "dbt test", "dbt source snapshot-freshness"],
            "settings": {"threads": 4, "target_name": "prod"},
            "state": 1,
            "schedule": {
                "date": {"type": "every_day"},
                "time": {"type": "every_hour", "interval": 1},
            },
        },
    )
    return future.result()["data"]
commented

@desertaxle I noticed after I updated prefect to prefect==2.0b9, that my orion server isn't working correctly anymore.

It doesn't show any of my flows that I verified work correctly.

Bela ran into the same issues yesterday too even after several fresh git clones using this example repo: https://github.com/sungchun12/prefect-dbt-cloud

Image

 ~/prefect-dbt-cloud │ main !1  python data_pipeline_demo.py                                                                                                                          ✔ │ 49s │ prefect-dbt-cloud Py 
15:08:57.139 | INFO    | prefect.engine - Created flow run 'neat-beagle' for flow 'Full Data Pipeline'
15:08:57.139 | INFO    | Flow run 'neat-beagle' - Using task runner 'ConcurrentTaskRunner'
15:08:57.184 | WARNING | Flow run 'neat-beagle' - No default storage is configured on the server. Results from this flow run will be stored in a temporary directory in its runtime environment.
15:08:57.322 | INFO    | Flow run 'neat-beagle' - Created task run 'Extract-bb1266fe-0' for task 'Extract'
15:08:57.411 | INFO    | Flow run 'neat-beagle' - Extract Raw Data
15:08:57.455 | INFO    | Task run 'Extract-bb1266fe-0' - Finished in state Completed()
15:08:57.501 | INFO    | Flow run 'neat-beagle' - Created task run 'Load-60b30268-0' for task 'Load'
15:08:57.560 | INFO    | Flow run 'neat-beagle' - Load Raw Data
15:08:57.595 | INFO    | Task run 'Load-60b30268-0' - Finished in state Completed()
15:08:57.630 | INFO    | Flow run 'neat-beagle' - Created task run 'Transform-a7d916b4-0' for task 'Transform'
15:08:57.677 | INFO    | Flow run 'neat-beagle' - Transform Raw Data
15:08:57.757 | INFO    | Task run 'Transform-a7d916b4-0' - Finished in state Completed()
15:08:57.823 | INFO    | Flow run 'neat-beagle' - Created task run 'Predict-f915ca64-0' for task 'Predict'
15:08:57.875 | INFO    | Flow run 'neat-beagle' - Predict results on transformed data
15:08:57.905 | INFO    | Task run 'Predict-f915ca64-0' - Finished in state Completed()
15:08:57.950 | INFO    | Flow run 'neat-beagle' - Finished in state Completed('All states completed.')
commented

@sungchun12 can you verify that
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api and rerun?

I use the following code:

from prefect import flow, task


@task
def shout(number):
    print(f"#{number}")


@flow
def count_to(highest_number):
    for number in range(1, highest_number + 1):
        if number % 2:
            shout.submit(number)
        else:
            shout(number)


if __name__ == "__main__":
    count_to(5)

Outputs:

15:22:23.856 | INFO    | prefect.engine - Created flow run 'adept-pegasus' for flow 'count-to'
15:22:23.856 | INFO    | Flow run 'adept-pegasus' - Using task runner 'ConcurrentTaskRunner'
15:22:23.866 | WARNING | Flow run 'adept-pegasus' - No default storage is configured on the server. Results from this flow run will be stored in a temporary directory in its runtime environment.
15:22:23.903 | INFO    | Flow run 'adept-pegasus' - Created task run 'shout-58a68b34-0' for task 'shout'
15:22:23.922 | INFO    | Flow run 'adept-pegasus' - Created task run 'shout-58a68b34-1' for task 'shout'
#1
#2
15:22:23.944 | INFO    | Task run 'shout-58a68b34-0' - Finished in state Completed()
15:22:23.950 | INFO    | Task run 'shout-58a68b34-1' - Finished in state Completed()
15:22:23.962 | INFO    | Flow run 'adept-pegasus' - Created task run 'shout-58a68b34-2' for task 'shout'
15:22:23.979 | INFO    | Flow run 'adept-pegasus' - Created task run 'shout-58a68b34-3' for task 'shout'
#3
#4
15:22:24.000 | INFO    | Task run 'shout-58a68b34-2' - Finished in state Completed()
15:22:24.008 | INFO    | Task run 'shout-58a68b34-3' - Finished in state Completed()
15:22:24.018 | INFO    | Flow run 'adept-pegasus' - Created task run 'shout-58a68b34-4' for task 'shout'
#5
15:22:24.043 | INFO    | Task run 'shout-58a68b34-4' - Finished in state Completed()
15:22:24.057 | INFO    | Flow run 'adept-pegasus' - Finished in state Completed('All states completed.')

image

Also can you try updating pip install -U "prefect>=2.0b"

commented

@belasobral93

Try this for async (add await and submit())

import asyncio
from prefect import task, flow


@task
async def trigger_dbt_cloud_job_run(job_id):
    await asyncio.sleep(2)
    return


@flow()
async def async_flow():
    print("job 1 called")
    await trigger_dbt_cloud_job_run.submit(64242)
    print("job 2 called")
    await trigger_dbt_cloud_job_run.submit(77213)
    print("job 3 called")
    await trigger_dbt_cloud_job_run.submit(77213)
    print("job 4 called")
    await trigger_dbt_cloud_job_run.submit(77213)


if __name__ == "__main__":
    asyncio.run(async_flow())

Should yield some async output


 `@flow(name='my_unique_name', ...)`
  warnings.warn(
17:01:33.645 | INFO    | prefect.engine - Created flow run 'dramatic-partridge' for flow 'async-flow'
17:01:33.645 | INFO    | Flow run 'dramatic-partridge' - Using task runner 'ConcurrentTaskRunner'
17:01:33.657 | WARNING | Flow run 'dramatic-partridge' - No default storage is configured on the server. Results from this flow run will be stored in a temporary directory in its runtime environment.
17:01:33.707 | INFO    | Flow run 'dramatic-partridge' - Created task run 'trigger_dbt_cloud_job_run-b91f6471-0' for task 'trigger_dbt_cloud_job_run'
17:01:33.741 | INFO    | Flow run 'dramatic-partridge' - Created task run 'trigger_dbt_cloud_job_run-b91f6471-1' for task 'trigger_dbt_cloud_job_run'
17:01:33.781 | INFO    | Flow run 'dramatic-partridge' - Created task run 'trigger_dbt_cloud_job_run-b91f6471-2' for task 'trigger_dbt_cloud_job_run'
17:01:33.798 | INFO    | Flow run 'dramatic-partridge' - Created task run 'trigger_dbt_cloud_job_run-b91f6471-3' for task 'trigger_dbt_cloud_job_run'
job 1 called
job 2 called
job 3 called
job 4 called
17:01:35.821 | INFO    | Task run 'trigger_dbt_cloud_job_run-b91f6471-1' - Finished in state Completed()
17:01:35.826 | INFO    | Task run 'trigger_dbt_cloud_job_run-b91f6471-0' - Finished in state Completed()
17:01:35.830 | INFO    | Task run 'trigger_dbt_cloud_job_run-b91f6471-2' - Finished in state Completed()
17:01:35.834 | INFO    | Task run 'trigger_dbt_cloud_job_run-b91f6471-3' - Finished in state Completed()
17:01:35.851 | INFO    | Flow run 'dramatic-partridge' - Finished in state Completed('All states completed.')
commented

@desertaxle

Getting different errors now after your suggestions.

Can you try running this example repo? https://github.com/sungchun12/prefect-dbt-cloud

Setup instructions are in the REAMDE

 ~/prefect-dbt-cloud │ main !1  python basic_flow.py            1 х │ 4s │ prefect-dbt-cloud Py 
Traceback (most recent call last):
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 61] Connect call failed ('127.0.0.1', 4200)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/prefect/client.py", line 374, in api_healthcheck
    await self._client.get("/health")
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/prefect/client.py", line 256, in send
    await super().send(*args, **kwargs)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "basic_flow.py", line 19, in <module>
    count_to(5)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/prefect/flows.py", line 367, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/prefect/engine.py", line 139, in enter_flow_run_engine_from_flow_call
    return anyio.run(begin_run)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/prefect/client.py", line 105, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/sung/prefect-dbt-cloud/env/lib/python3.8/site-packages/prefect/engine.py", line 179, in create_then_begin_flow_run
    raise RuntimeError(
RuntimeError: Cannot create flow run. Failed to reach API at http://127.0.0.1:4200/api/.

 ~/prefect-dbt-cloud │ main !1  prefect orion start               ✔ │ 6s │ prefect-dbt-cloud Py 
Starting...

 ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
| _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
|  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
|_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api

Check out the dashboard at http://127.0.0.1:4200



INFO:     Started server process [82513]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/base.py", line 227, in _catch_revision_errors
    yield
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/base.py", line 439, in _upgrade_revs
    for script in reversed(list(revs))
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/revision.py", line 789, in iterate_revisions
    revisions, heads = fn(
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/revision.py", line 1416, in _collect_upgrade_revisions
    current_revisions = self.get_revisions(lower)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/revision.py", line 525, in get_revisions
    return sum([self.get_revisions(id_elem) for id_elem in id_], ())
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/revision.py", line 525, in <listcomp>
    return sum([self.get_revisions(id_elem) for id_elem in id_], ())
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/revision.py", line 549, in get_revisions
    return tuple(
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/revision.py", line 550, in <genexpr>
    self._revision_for_ident(rev_id, branch_label)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/revision.py", line 619, in _revision_for_ident
    raise ResolutionError(
alembic.script.revision.ResolutionError: No such revision or branch '628a873f0d1a'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/starlette/routing.py", line 635, in lifespan
    async with self.lifespan_context(app):
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/starlette/routing.py", line 530, in __aenter__
    await self._router.startup()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/starlette/routing.py", line 612, in startup
    await handler()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/orion/api/server.py", line 259, in run_migrations
    await db.create_db()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/orion/database/interface.py", line 53, in create_db
    await self.run_migrations_upgrade()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/orion/database/interface.py", line 61, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 54, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/orion/database/alembic_commands.py", line 29, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/orion/database/migrations/env.py", line 98, in <module>
    apply_migrations()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 118, in wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 65, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/orion/database/migrations/env.py", line 92, in apply_migrations
    await connection.run_sync(do_run_migrations)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/engine.py", line 546, in run_sync
    return await greenlet_spawn(fn, conn, *arg, **kw)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 128, in greenlet_spawn
    result = context.switch(value)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/prefect/orion/database/migrations/env.py", line 80, in do_run_migrations
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/runtime/migration.py", line 608, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/command.py", line 309, in upgrade
    return script._upgrade_revs(revision, rev)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/base.py", line 435, in _upgrade_revs
    return [
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/sung/.pyenv/versions/3.8.9/lib/python3.8/site-packages/alembic/script/base.py", line 259, in _catch_revision_errors
    raise util.CommandError(resolution) from re
alembic.util.exc.CommandError: Can't locate revision identified by '628a873f0d1a'

ERROR:    Application startup failed. Exiting.
Orion stopped!
commented

alembic.util.exc.CommandError: Can't locate revision identified by '628a873f0d1a'

I think there was a breaking change in the database; the easiest way to migrate is to delete ~/.prefect/orion.db

commented

@ahuang11 Still getting the same errors with the same error message after deleting this multiple times: ~/.prefect/orion.db

commented

Can you try updating pip install "prefect>=2.0b"?

commented

@sungchun12 I ran into a similar issue recently and I fixed it by recreating my virtual environment. We can also hop on a call together if that would help resolve issues more quickly.

commented

@desertaxle Thanks for the offer. Let's hope on a quick call: https://calendly.com/d/dvr-zt7-psg/30-minute-meeting

commented

@sungchun12 The earliest available time I saw was Tuesday morning so that's when I booked time, but let me know if you'd like to meet sooner.

commented

@desertaxle saw the meeting invite come through, Tuesday will have to do thanks!

commented

@ahuang11 @desertaxle Looks like there may be something wrong with the create_flow_run() that impacts trigger_dbt_cloud_job_run(). Running the example flow from docs produces the following (same 422 error when I try the async code):

Code

from prefect import flow


@flow(name="Hello Flow")
def hello_world(name="world"):
    print(f"Hello {name}!")


hello_world("Marvin")

Logs

Traceback (most recent call last):
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/async_flow.py", line 17, in <module>
    hello_world("Marvin")
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/flows.py", line 367, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 134, in enter_flow_run_engine_from_flow_call
    return anyio.run(begin_run)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/belaloaner/.pyenv/versions/3.10.4/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/belaloaner/.pyenv/versions/3.10.4/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/client.py", line 107, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 188, in create_then_begin_flow_run
    flow_run = await client.create_flow_run(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/client.py", line 604, in create_flow_run
    response = await self._client.post("/flow_runs/", json=flow_run_create_json)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/client.py", line 280, in send
    response.raise_for_status()
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/client.py", line 226, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url 'https://api-beta.prefect.io/api/accounts/6941ed01-3570-4537-92b5-79296385f7c0/workspaces/3674ba46-c9af-4827-858c-9fedbf830913/flow_runs/'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'loc': ['body', 'flow_runner'], 'msg': 'extra fields not permitted', 'type': 'value_error.extra'}], 'request_body': {'name': 'nocturnal-bullfinch', 'flow_id': 'f10f715f-512b-4226-b177-680f13805858', 'deployment_id': None, 'flow_version': '6c3a0e3ef179a52c083eb85997507207', 'parameters': {'name': 'Marvin'}, 'idempotency_key': None, 'context': {}, 'empirical_policy': {'max_retries': 0, 'retry_delay_seconds': 0.0}, 'tags': [], 'parent_task_run_id': None, 'flow_runner': None, 'state': {'type': 'PENDING', 'name': 'Pending', 'message': None, 'data': None, 'state_details': {'flow_run_id': None, 'task_run_id': None, 'child_flow_run_id': None, 'scheduled_time': None, 'cache_key': None, 'cache_expiration': None}}}}
For more information check: https://httpstatuses.com/422
commented

@belasobral93 Can you try upgrading prefect with pip install "prefect>=2.0b12"?

commented

thanks @ahuang11 that worked and I see async behavior. I have a question - how can I call the same prefect task with different conditions? Do they need to be in sub-flows?

For example, I want to call trigger_dbt_cloud_job_run() for 2 different jobs, but I want each of them to have different delays (sleep(5), sleep(2). I can't create different tasks for each one, because I can't call a task within a task. So is my only option to put these tasks inside flows? Or can I somehow give these tasks aliases?

I want to set different delays for the following task:

@task
async def trigger_dbt_cloud_job_run(job_id):
    await asyncio.sleep(5) 
    return
commented

  • Verify I can plug and play an API token, account id, job id into python code with minimal python knowledge

  • Verify the prefect function docstrings are plainly understood from a data analyst perspective

  • Verify I can trigger a job synchronously with polling

  • Verify I can get any artifact such as sources.json, run_results.json, catalog.json, manifest.json, example_sql.sql

  • Verify I can send custom trigger options such a step override commands from a previous task dynamically

  • Verify I can trigger a job async

[👩🏽‍💻] Verify I can create a dbt Cloud job using an arbitrary config with json

[❌] Verify I get a dbt Cloud job to succeed with nice hyperlinks and logs

  • @ahuang11 @desertaxle Need the final status saying the job has status SUCCESS & run data returned from dbt's api like trigger_dbt_cloud_job_run() does.
4:15:02.165 | INFO    | Flow run 'eccentric-urchin' - dbt Cloud job run with ID 70820362 has status RUNNING. Waiting for 5 seconds.
14:15:07.358 | INFO    | Flow run 'eccentric-urchin' - Created task run 'Get dbt Cloud job run details-46ef67b7-12' for task 'Get dbt Cloud job run details'
14:15:07.358 | INFO    | Flow run 'eccentric-urchin' - Executing 'Get dbt Cloud job run details-46ef67b7-12' immediately...
14:15:08.521 | INFO    | Task run 'Get dbt Cloud job run details-46ef67b7-12' - Finished in state Completed()
14:15:08.770 | INFO    | Flow run 'eccentric-urchin' - Created task run 'List dbt Cloud job artifacts-fabe6c19-0' for task 'List dbt Cloud job artifacts'
14:15:08.771 | INFO    | Flow run 'eccentric-urchin' - Executing 'List dbt Cloud job artifacts-fabe6c19-0' immediately...
14:15:10.253 | INFO    | Task run 'List dbt Cloud job artifacts-fabe6c19-0' - Finished in state Completed()
14:15:10.414 | INFO    | Flow run 'eccentric-urchin' - Finished in state Completed()
(env) belaloaner@bela-loaner prefect-dbt-cloud_2 %
  • Verify I get a dbt Cloud job to fail with helpful hyperlinks and logs
    • logs clearly show that flow failed due to an error in a specific dbt cloud run - run is hyperlinked.
commented

@belasobral93 I don't quite understand, but here's what I think!

@task
async def trigger_dbt_cloud_job_run_one(job_id):
    await asyncio.sleep(5) 
    return

@task
async def trigger_dbt_cloud_job_run_two(job_id):
    await asyncio.sleep(2) 
    return

@flow 
async def trigger_flow():
    task_one = trigger_dbt_cloud_job_run_one()
    task_two = trigger_dbt_cloud_job_run_two()

Or if you must call task within task

@task
async def trigger_dbt_cloud_job_runs():
    trigger_dbt_cloud_job_run.fn()  # call the underlying Python function so it's not a task
    trigger_dbt_cloud_job_run.fn()  # call the underlying Python function so it's not a task

You can also rename tasks like
trigger_dbt_cloud_job_run.with_options(name="trigger_one")

commented

@ahuang11

@task
async def trigger_dbt_cloud_job_runs():
    trigger_dbt_cloud_job_run.fn()  # call the underlying Python function so it's not a task
    trigger_dbt_cloud_job_run.fn()  # call the underlying Python function so it's not a task

^^ Is what I was looking for. If I call trigger_dbt_cloud_job_run_one() it won't know that I want to call trigger_dbt_cloud_job_run() specifically.

If I use (name="trigger_one") does that mean I could then call it like this?

@task
async def trigger_one():
commented

If I use (name="trigger_one") does that mean I could then call it like this?

No I don't think so; it just shows up in the UI/terminal with that name, e.g. https://discourse.prefect.io/t/how-to-display-more-descriptive-task-run-names-in-the-radar-view-ui/1146

You could certainly wrap it inside another python function, but I don't quite understand the goal.

commented

[❌] Verify I can create a dbt Cloud job using an arbitrary config with json
Getting an error using docstrings example.

To try and resolve I did the following (was not successful in resolving):

  • removed the trailing comma after the schedule object (I believe trailing commas not allowed in json?)
  • Compared and validated body structure + included all elements listed in our postman collection for POST Create Job
14:55:27.498 | ERROR   | Flow run 'upbeat-moose' - Finished in state Failed('Flow run encountered an exception.')
Traceback (most recent call last):
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/arbitrary.py", line 54, in <module>
    create_job_flow()
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/flows.py", line 367, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 150, in enter_flow_run_engine_from_flow_call
    return anyio.run(begin_run)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/belaloaner/.pyenv/versions/3.10.4/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/belaloaner/.pyenv/versions/3.10.4/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 226, in create_then_begin_flow_run
    return state.result()
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 559, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/arbitrary.py", line 14, in create_job_flow
    future = call_dbt_cloud_administrative_api_endpoint(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/tasks.py", line 282, in __call__
    return enter_task_run_engine(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 688, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 136, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/Users/belaloaner/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/Users/belaloaner/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 803, in create_task_run_then_submit
    return await future._result()
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/futures.py", line 220, in _result
    return final_state.result(raise_on_failure=raise_on_failure)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect/engine.py", line 1039, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect_dbt/cloud/utils.py", line 113, in call_dbt_cloud_administrative_api_endpoint
    response = await client.call_endpoint(
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/prefect_dbt/cloud/clients.py", line 56, in call_endpoint
    response.raise_for_status()
  File "/Users/belaloaner/Desktop/prefect-dbt-cloud_2/env/lib/python3.10/site-packages/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://cloud.getdbt.com/api/v2/accounts/43799/jobs/'
For more information check: https://httpstatuses.com/400
commented

@belasobral93 For the prefect Orion UI, Alex and I were able to make it work but through non-repeatable means because prefect 2.0 is still in flux. Once prefect 2.0 is GA tomorrow, the UI errors should decrease.

With that in mind, once all the mechanical pieces above are done, we'll have prefect GA this integration tomorrow so that broader adoption can happen and fix further bugs.

After, you should be ready to tackle #3 and #7

commented

@belasobral93 I updated the create job example in #37 and it should be working now. Also, trigger_dbt_cloud_job_run_and_wait_for_completion returns the result of the job run along with the artifacts. If you print out the return value from the flow, you can inspect it to make sure that it has all of the fields that you are expecting. Here's an output that I got:

{
   "id":71164019,
   "trigger_id":71879132,
   "account_id":36394,
   "environment_id":97658,
   "project_id":118320,
   "job_definition_id":97588,
   "status":10,
   "dbt_version":"1.1.0-latest",
   "git_branch":"demo-sung",
   "git_sha":"6b86c1e47900b3caa37a2af840770faa1c4ef2b2",
   "status_message":"None",
   "owner_thread_id":"None",
   "executed_by_thread_id":"dbt-run-71164019",
   "deferring_run_id":"None",
   "artifacts_saved":true,
   "artifact_s3_path":"prod/runs/71164019/artifacts/target",
   "has_docs_generated":true,
   "has_sources_generated":true,
   "notifications_sent":true,
   "blocked_by":[
      
   ],
   "scribe_enabled":true,
   "created_at":"2022-07-27 12:30:03.850957+00:00",
   "updated_at":"2022-07-27 12:32:03.363227+00:00",
   "dequeued_at":"2022-07-27 12:30:05.318124+00:00",
   "started_at":"2022-07-27 12:30:11.271817+00:00",
   "finished_at":"2022-07-27 12:32:02.950524+00:00",
   "last_checked_at":"2022-07-27 12:32:03.251493+00:00",
   "last_heartbeat_at":"2022-07-27 12:31:41.302098+00:00",
   "should_start_at":"2022-07-27 12:30:03.850957+00:00",
   "trigger":"None",
   "job":"None",
   "environment":"None",
   "run_steps":[
      
   ],
   "status_humanized":"Success",
   "in_progress":false,
   "is_complete":true,
   "is_success":true,
   "is_error":false,
   "is_cancelled":false,
   "href":"https://cloud.getdbt.com/#/accounts/36394/projects/118320/runs/71164019/",
   "duration":"00:01:59",
   "queued_duration":"00:00:07",
   "run_duration":"00:01:51",
   "duration_humanized":"1 minute, 59 seconds",
   "queued_duration_humanized":"7 seconds",
   "run_duration_humanized":"1 minute, 51 seconds",
   "created_at_humanized":"2 minutes, 5 seconds ago",
   "finished_at_humanized":"6 seconds ago",
   "job_id":97588,
   "is_running":"None",
   "artifact_paths":[
      ...
   ]
}

I truncated artifact_paths for readability, but is that what you're expecting to see?

commented

(checkboxes for CLI, will do for UI)

  • Verify I can plug and play an API token, account id, job id into python code with minimal python knowledge

  • Verify the prefect function docstrings are plainly understood from a data analyst perspective

  • Verify I can trigger a job synchronously with polling

  • Verify I can get any artifact such as sources.json, run_results.json, catalog.json, manifest.json, example_sql.sql

  • Verify I can send custom trigger options such a step override commands from a previous task dynamically

  • Verify I can trigger a job async

  • Verify I can create a dbt Cloud job using an arbitrary config with json

  • Verify I get a dbt Cloud job to fail with helpful hyperlinks and logs

  • Verify I get a dbt Cloud job to succeed with nice hyperlinks and logs
    @desertaxle great, that data response is what I was looking for. Can we get also get a cloud job run status printed to log when a run is successful?

commented

@desertaxle @belasobral93 What's left to get this officially released? Just that missing checkbox in the previous comment?

commented

@sungchun12 yep! Nothing left on my end other than that ask

commented

I opened #40 to add a log message letting the user know that a run was successful. @belasobral93 can you confirm that this fulfills the last requirement? If so, we should be able to get #40 merged and release prefect-dbt today!

commented

@desertaxle it does, thank you. We're good to go @sungchun12

commented

Thanks for the progress on this. I’m honestly so excited to brag about how fun and meaningful this work has been together. I already said this a couple times in candid conversations, but this is a great role model for what open source, async partnership can look and feel like. And it feels quite splendid!

commented

@belasobral93 you should make a quick and fun “QuickStart” tutorial this week in loom! I’ll share it with pride :)

commented

I'm going to close this issue since it sounds like we've finished QA. I've opened #41 to prepare for the first release of prefect-dbt, so we can discuss anything that's needed prior to release there!

commented