MLOps Mini-Course 2022

This is a compact but all-encompassing set of workshops to practice cloud-native MLOps tools like Kubeflow, Pachyderm, Feast, and Seldon. All this experiments are intended to run in k8s environment. It's recommended to use Minikube deployed on Linux server (minimum 12 GB RAM, 2 CPU, 50 GB of disk space).

Environment setup

Launch a server or use your own computer if it satisfies the aforementioned requirements.
Install Docker. Since we'll deploy a lot of things, make sure that you increased the max limit for open file descriptors.
Install kubectl
Install kustomize 3.2.0
Install helm
Install Minikube. You will also need to enable Docker to run as a non-root user
Start a local k8s cluster (the following k8s version since Kubeflow is not yet compatible with latest k8s versions):
```
minikube start --kubernetes-version=v1.20.2 --memory 30000 --cpus 4
```
Verify that it works by checking available pods. Also you can launch a dashboard:
```
kubectl get po --all-namespaces
minikube dashboard
```

Now let's install Kubeflow. Clone the official repo with manifests and install them:

while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

Also, let's use the same repo to install Feast manifests and Seldon manifests
Verify installation by proxying istio gateway to local port and by checking pods within your cluster. You can login to the Kubeflow dashboard with the following credentials - user@example.com / 12341234
Also, we'll need to install Pachyderm locally. Use this installation guide. However, it's preferred to create a separate namespace for the service (e.g. pachdyderm).

Kubernetes basics

Visit the kubernetes directory and deploy a basic application with Service and HPA:

kubectl apply -f .

Tensorboard

Create a new Kubeflow Notebook (based on pytorch) and run this sample notebook. Afterwards, you can go to Kubeflow Tensorboard and create a new server with a PVC of the corresponding notebook pointing to the directory with logs.

Pachyderm

Considering that you have Pachyderm installed and configured, you can run the starter notebook.

After that, go to the housing-prices directory to explore how Pachyderm can be used for model training and deployment.

You can also enable Promethus monitoring by using the following tutorial

Feast

Open Kubeflow dashboard, create a new notebook. In the notebook clone this repo. Navigate to feast directory. Current code will use local files and registry, but it can be replaced with s3 or gcs. Run the notebook to see how Feast provides ability to query features both in the offline and online modes.

Katib (hyper-parameter optimization)

Get familiar with Katib.

Go to katib directory and apply the basic mnist experiment:

    kubectl apply -f .

After than you can navigate to Katib tab in the Kubeflow dashboard and check the newly created experiment.

Code for the coresponding example.

KFP (Kubeflow Pipelines)

Navigate to the directory with a sample pipeline. It's integrated with Feast and Pachyderm services that were used before. You don't need to build docker image - it's already available in my public docker registry. You can run the pipeline (don't forget to port-forward ml-pipeline-ui service and change the url to localhost) or just compile it and upload it manually afterwards (disregard connection error - pipeline archive will be generated anyway):

python pipeline.py

Kubeflow Kale

There's a sample Kale notebook that generates a KFP pipeline for Titanic dataset classification. In order to run it:

1. Create a new Kubeflow notebook with a CUSTOM image: `gcr.io/arrikto/jupyter-kale:v0.5.0`

2. Copy the contents of the Kale demo directory to the notebook

3. Open the titanic_dataset_ml.ipynb file there and activate the Kale jupyter extension on the left bar.

4. Change the name of your mounted volume so it's used in the pipeline

5. Click "Compile and Upload" button

6. Navigate to KFP dashboard in the browser and run the pipeline

Oleiva / mlops_workshops