This repository contains common and extensible Prefect patterns to drive efficient workflows β we like to call these patterns our recipes
Here you'll find starter code and more advanced example use cases.
We're always looking for new contributions! See our existing Recipe Ideas/Issues for inspiration. Read a detailed guide on how to share your solutions with the Prefect community or run these commands to get started right away.
To report issues, typos, or link fixes, please open an issue. We appreciate it!
What are you interested in seeing examples of? Jot down your big idea here.
- Getting Started With Subflows
- Run Multiple Subflows or Child Flows in Parallel
- Subflow with a Different Task Runner Than Parent Flow
- Create a Subflow and Block Until It's Completed
- Running Subflows On Their Own Infrastructure Using a Separate Deployment
- Conditionally Stop a Task Run
- Ensure Tasks Immediately Fail If Upstream Task Fails
- Define State Dependencies Between Tasks
- Triggering Flow Runs From a Deployment via API Call or From a Terminal Using Curl
- Event-Driven Data Pipelines with AWS Lambda and GitHub Actions
- Deploy Flows to Run as a Local Process, Docker Container or a Kubernetes Job
- Deploy Flows to AWS
- Deploy Flows to GCP
- Deploy Flows to Azure
- Store Flows in GitHub and Execute in a Docker Container
- Python-Based Deployments
- Scheduled Data Pipelines in 5 Minutes with Prefect and GitHub Actions - Blog
- Declarative Dataflow Deployments with Prefect Make CI/CD a Breeze
- Deploy Prefect Pipelines with Python: Perfect!
- Prefect Deployments FAQ (PDF) - DataflowOps for Prefect 2.0
- GCP and Prefect Cloud β from Docker Container to Cloud VM on Google Compute Engine
- Deployment patterns & examples: Code examples showing flow deployment to various types of infrastructure - GitHub Repo
- Docker: How to use Prefect Deployments with flow code baked into a Docker image - GitHub Repo
- AWS Fargate: Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate - GitHub Repo
- AWS EKS: Project demonstrating how to automate Prefect 2.0 deployments to AWS EKS - GitHub Repo
- Event-driven Data Pipelines with AWS Lambda, Prefect and GitHub Actions
- Scheduled vs. Event-driven Data Pipelines β Orchestrate Anything with Prefect
- You No Longer Need Two Separate Systems for Batch Processing and Streaming
- GCP: Serverless Prefect Flows with Google Cloud Run Jobs
- Azure: Serverless Prefect Flows with Azure Container Instances
- AWS: Prefect + AWS ECS Fargate + GitHub Actions Make Serverless Dataflows As Easy as .py
- AWS Lambda: Deploy a Prefect flow to serverless AWS Lambda function - GitHub Repo
- Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions
- Build a Data Platform with Prefect, dbt, and Snowflake (using blocks)
- Real World Python for Data Engineering - Supercharge Your Data Orchestration with Prefect 2.0
- Create a Maintainable Data Pipeline with Prefect and DVC
- Orchestrating Airbyte with Prefect 2
- Coordinate ELT in 2023 with Airbyte, dbt and Prefect
- Schedule & orchestrate dbt Cloud jobs with Prefect
- Prefect & Fivetran: integrate all the tools & orchestrate them in Python
- Export Airbyte Configuration and Load to S3 bucket using blocks, including Python-based deployment
- ELT with Snowflake Using Async and Blocks
- Deploy a Prefect agent to ECS using the AWS CLI
- Deploy a Prefect agent to ECS with Terraform
- Deploy Flows Using ECSTask Infrastructure Blocks
- Deploy a Prefect agent to ECS Fargate using CloudFormation and GitHub Actions
- Setup Azure with Prefect
- Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
- Setup an Azure VM and Run the Prefect Agent
- Deploy Flows Using Azure Container Instances Infrastructure Blocks
- How to use Kubernetes with Prefect
- How to use Kubernetes with Prefect: Part 2
- How to use Kubernetes with Prefect: Part 3
- Build flow image and Prefect deployment with storage and infra Blocks on push to branch
- Build and Push flow docker image to Google Artifact Registry
- Build / Apply prefect deployment with blocks on change to python files
- Automate Prefect Deployments to AWS ECS Fargate Using GitHub Actions
- Automate Python-Based Deployments with GitHub Actions
- Create Observable and Reproducible Notebooks with Hex - Article
- Create Observable and Reproducible Notebooks with Hex - Video
- (Re)Introducing Prefect: The Global Coordination Plane
- The Dataflow Coordination Spectrum
- Why Prefect?
- Workflow Orchestration vs. Data Orchestration β Are Those Different?
- Dataflow Design Patterns
- How to Structure an ML Project for Reproducibility and Maintainability
- Orchestrate Your Data Science Project with Prefect 2.0
- Build a Full-Stack ML Application With Pydantic And Prefect
- How to Build a Modular Data Stack β Data Platform with Prefect, dbt and Snowflake
- How to Build Modular Dataflows with Tasks, Flows and Subflows in Prefect
- Register a Prefect Flow
- Run GraphQL Queries
- Airbyte Orchestration
- ETL with AWS S3 and Snowflake
- Use AWS Lambda for Event-Driven Flows
- Read Secrets into Prefect Cloud tenant
- Handle DBT Model Failures
- S3 Flow Storage on EKS
- Use LocalExecutor to run Dask computations on a Coiled cluster
- Use DaskExecutor to run Prefect tasks in parallel on a Coiled cluster
We're always looking for new contributions! You can add your Prefect 2.0 recipe and earn some swag in a few simple steps:
- Clone the prefect-recipes repo:
git clone git@github.com:PrefectHQ/prefect-recipes.git
- Create and checkout a new branch:
git checkout -b feat/new-recipe-name
- Add your code under the appropriate category. Unsure? Add it under
flows-advanced/
. - Add your recipe to this README.
- Commit and push the code to your remote branch.
- Create a PR π€
We use our Slack Community and Discourse to discuss all things Prefect-- such as FAQ, use cases and integrations. Join in the conversation π
Thank you for your contributions and efforts to improve prefect-recipes. We're glad to have you in our community!