peeush-agarwal / prefect-learn

Prefect (Model orchestration tool) learning resources

prefect-learn

Prefect (Model orchestration tool) learning resources.

Reference

Get started

1. Create conda virtual environment: conda env create -f requirements.yaml
2. Activate the conda environment: conda activate prefect-env
3. Good to go!

Tutorials

1. First steps
2. Flow and Task Configuration
3. Task dependencies
4. Flow execution
   1. Task runners and Parallel execution
   2. Asynchronous execution
5. Dask and Ray task runners
   1. Sequential task runner
   2. Dask task runner
   3. Ray task runner
   4. Ray subflow task runner

Deployment tutorial

1. Create a script with the flow. leo.py
2. Create a deployment specification for this flow. leo_deployment.py
3. Set the Prefect API server URL: prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
4. Configure storage: prefect storage create
   1. Make sure the bucket already exists in S3 buckets.
5. Create the deployment: prefect deployment create leo_deployment.py
6. List all of the current deployments: prefect deployment ls
7. Display details of the specific deployment: prefect deployment inspect 'leonardo_dicaprio_flow/leonardo-deployment'
8. Run the deployment locally: prefect deployment execute leonardo_dicaprio_flow/leonardo-deployment
   1. Format of the name: flow_name/deployment_name
   2. Flow may be referenced by multiple deployments, each deployment must have a unique name.
9. View the deployment in Prefect UI: Deployments
10. Create a work queue: prefect work-queue create --tag tutorial tutorial_queue
    1. Note that this command specifically creates a "tutorial" tag on the work queue, meaning the tutorial_queue work queue will only serve deployments that include a "tutorial" tag. This is a good practice to make sure flow runs for a given deployment run only in the correct environments, based on the tags you apply.
    2. The Prefect API creates the work queue and returns the ID of the queue. Note this ID, you'll use it in a moment to create an agent that polls for work from this queue.
11. List of available work queues: prefect work-queue ls
12. Run agent: prefect agent start 'c86c2f96-f366-498c-acee-d9c8e310ec52'

Remember that:

1. The deployment specification included a "tutorial" tag.
2. The tutorial_queue work queue is defined to serve deployments with a "tutorial" tag.
3. The agent is configured to pick up work from tutorial_queue, so it will only execute flow runs for deployments with a "tutorial" tag.

Run an orchestrated deployment

With a work flow and agent in place, you can create a flow run for leonardo_dicaprio_flow directly from the UI. Go back to the Prefect UI in your browser and click the Quick Run button next the deployment.

Add another deployment

1. Update leo_deployment.py with another DeploymentSpec
2. Create the deployment: prefect deployment create leo_deployment.py
   1. It won't change leonardo_deployment, but it will create a new marvin-deployment.
      1. Running leonardo_deployment logs the message "Hello Leo!".
      2. Running marvin-deployment logs the message "Hello Marvin!".
      3. marvin-deployment uses the SubprocessFlowRunner.
3. Both deployments can use the tutorial_queue work queue because they have "tutorial" tags. But if you created a new work queue that served deployments with, say, a "dev" tag, only marvin-deployment would be served by that queue and its agents.

Cleaning up

1. To terminate the agent, simply go to the terminal session where it's running and end the process with either Ctrl+C or by terminating the terminal session.
2. To pause a work queue, run the Prefect CLI command prefect work-queue pause, passing the work queue ID.
3. To delete a work queue, run the command prefect work-queue delete, passing the work queue ID.
4. To terminate the Prefect API server, go to the terminal session where it's running and end the process with either Ctrl+C or by terminating the terminal session.