nogsantos / poc-prefect

Workflow orchestration - https://www.prefect.io/ - https://docs.prefect.io/core/about_prefect/why-prefect.html

Poc Prefect

• Docs: https://docs.prefect.io

Requirements

• Python 3.6.x
• Docker

Local Setup

Project dependencies

pipenv install

Prefect viz module

• Tool - installing-optional-dependencies

Tools for visualizing Prefect flows.

pipenv run pip install 'prefect[viz]'

Localhost development

Init prefect local backend server

pipenv run prefect backend server

Prefect UI init

pipenv run prefect server start

Create a project

pipenv run prefect create project "Tests"

Registering a task

pipenv run prefect register -p ./etl/funcional_flow.py --project "Tests"

Setup Universal deployment

• Docs - universal-deploy

Init prefect universal backend cloud

pipenv run prefect backend cloud

Autenticate

pipenv run prefect auth login --key <key>

Create a project

pipenv run prefect create project <projct-name>

Register the flow

As an example, using the flow funcional_flow.py to create in https://cloud.prefect.io/

pipenv run prefect register -p ./etl/funcional_flow.py --project <project-name>

Start the agent

pipenv run prefect agent <agent-name> start