Flyte Demo

aircraft etl example using Flyte, adapted from the Prefect Tutorial Aircraft ETL Example

Getting started

Prerequisites:

make
node (required for pyright)
python >= 3.10
docker

To start:

Install the development environment: make install
Install flytectl: make flytectl

Flyte Usage

Once the virtualenv has been created and activated, you can run the worklow locally, eg:

make run

To run in the Flyte sandbox:

## Start demo sandbox, mounting the current dir (ie: this repo)
make sandbox

## Build the docker image and push it to the sandbox
make build

## Deploy, ie: register and execute
make deploy

## Visualise the execution graph
make viz

## Monitor ie: get all executions
make ge

Interactive commands (eg: ash) won't work properly inside the sandbox (you don't get their output). But you can run non-interactive commands, eg:

flytectl sandbox exec -- ls

Accessing kubes:

# set kubeconfig (if not using direnv)
export KUBECONFIG=$HOME/.flyte/config-sandbox.yaml

# set default namespace to aid debugging
kubectl config set-context --current --namespace=flytesnacks-development

UI

Flyte console: http://localhost:30080/console
Minio console: http://localhost:30002/. User: minio pass: miniostorage.
Kubes dashboard: http://localhost:30080/kubernetes-dashboard

FAQ

What is the distinction between flytectl sandbox and flytectl demo

Troubleshooting

task with different structure already exists

Existing tasks cannot be deleted. So instead, register the task with a new version. Version strings are arbitrary.

Last Error: USER::containers with unready status: ... Back-off pulling image "aircraft:latest"

Don't use the tag latest because Kubes will try and pull, but the image won't exist. See Error: ImagePullBackOff.

Last Error: USER::

If the pods have been OOMKilled then increase the kubernetes memory limits on the task, or up the defaults in infra/config.yaml

tekumara / flyte-demo