tekumara / prefect-demo

Prefect examples running in a local kubernetes cluster

Prefect Demo

Prefect 2 (aka Orion) examples running self-contained in a local kubernetes cluster. Batteries (mostly) included. 🔋

Getting started

Prerequisites:

• make
• node (required for pyright)
• python >= 3.10
• docker & docker compose
• k3d (for creating a local kubernetes cluster)
• kubectl
• helm

To start:

• Install the development environment: make install

Examples

Flows:

• Dask flow using a dask local cluster
• Dask kubes flow using an ephemeral dask cluster on Kubernetes
• Parameterized flow using a custom Docker image containing additional modules
• Ray flow that runs on an existing ray cluster (see tekumara/ray-demo)
• Sub flow that is trigger by a parent flow
• Submit flow demonstrates the difference of running tasks with/without .submit()
• Retry flow demonstrates retries when a task fails.
• Failure flow shows how to handle a failing task.
• Map flow uses Task.map.
• Flatten flow demonstrates how to parallelise over a list of lists.
• Parent flow shows how to trigger a flow run from a deployment within a parent flow. The triggered flow is treated as a sub flow.
• Context flow accessing prefect context at runtime.

Deployments to Kubernetes created via:

• Deployment objects in python - both local and uploaded to remote s3 storage.
• prefect.yaml - both local and pushed to s3.

Usage

Local

1. make param-flow or make dask-flow or make ray-flow or make sub-flow
2. make ui then navigate to http://localhost:4200/

The orion sqlite database is stored in ~/.prefect/prefect.db

Kubernetes

Create k3d cluster with an image registry, minio (for remote storage), the prefect agent and api

make kubes

Create deployments that run on kubernetes

make deploy

UI

Prefect UI: http://localhost:4200/

Minio UI: http://localhost:9001. User: minioadmin pass: minioadmin.

API

Prefect API: http://localhost:4200/api/

Docs

• Blocks - an overview and look into the database tables for Blocks.
• Deployment - an overview of the deployment process.

References

• Tutorials from which some of the examples are taken

Cloud

To run flows with a cloud workspace set:

export PREFECT_API_URL=https://api.prefect.cloud/api/accounts/$accountid/workspaces/$workspaceid
export PREFECT_API_KEY=<your api key>

$accountid and $workspaceid are visible in the URL when you login to Prefect Cloud. The api key can be created from your user profile (bottom left).

Setting the environment variables is recommended. An alternative method is to login using:

prefect cloud login --key <your api key>

However be aware that this stores your api url and key as plain text ~/.prefect/profiles.toml.

Ray

Create a kubernetes ray cluster

make kubes-ray

Ray dashboard: http://localhost:8265

Known issues

Major

• Support flat mapping #6462
• Zombie tasks consume concurrency slots

Minor

• Replicate kube events to Prefect that occur before the pod starts
• Configuring different task runner for dev and prod deployments
• Workers only support local storage
• Prefect.Deployment doesn't have a property pull_steps
• Support environment variables read from kubernetes secrets
• Encountered error while running prefect.deployments.steps.set_working_directory - FileNotFoundError
• Deployment metadata
• Stream logs via CLI
• FileNotFoundError errors when running with a remote ray cluster #26
• Add Flow.submit interface for subflows on external infrastructure
• not a known member of module "prefect.runtime"
• mapped tasks aren't collapsed in the UI like they were in Prefect 1
• pushing to s3 is sequential and slower than using sf3s to parallel upload to an s3 storage block

See all roadmap tagged issues for planned work.

Troubleshooting

Flows are late

Check the logs of the agent/worker:

make kubes-logs