yankcrime / deployKF

deployKF effortlessly integrates Kubeflow and leading MLOps tools on Kubernetes, compose your open ML platform today!

deployKF

Deploy the Machine Learning Toolkit for Kubernetes

deployKF effortlessly integrates Kubeflow and leading MLOps tools on Kubernetes, compose your open ML platform today!

_{Crafted with ❤️ by the developers of Kubeflow.}

About

What is deployKF?

deployKF is the best way to build reliable ML Platforms on Kubernetes.

• deployKF supports leading MLOps & Data tools from both Kubeflow, and other projects
• deployKF has a Helm-like interface, with values for configuring all aspects of the deployment (no need to edit Kubernetes YAML)
• deployKF does NOT install resources directly in your cluster, instead it generates ArgoCD Applications to provide native GitOps support

What tools does deployKF support?

deployKF currently supports MLOps tools from the Kubeflow ecosystem like Kubeflow Pipelines and Kubeflow Notebooks, for the full list of current tools, please see the supported tools page.

We are actively adding support for other popular tools such as MLflow Model Registry, Apache Airflow, and Feast. For a more complete list of planned tools, please see the future tools page.

Who makes deployKF?

deployKF was originally created by Mathew Wicks (GitHub: @thesuperzapper), a Kubeflow lead and maintainer of the popular Apache Airflow Helm Chart. However, deployKF is now a community-led project that welcomes contributions from anyone who wants to help.

For commercial services related to deployKF, please see the support page.

Who uses deployKF?

deployKF is a new project, and we are still building our community. If you are using deployKF, please consider adding your organization to our list of adopters.

What is the difference between Kubeflow and deployKF?

Kubeflow and deployKF are two different but related projects:

• deployKF is a tool for deploying Kubeflow and other MLOps tools on Kubernetes as a cohesive platform.
• Kubeflow is a project that develops MLOps tools, including Kubeflow Pipelines, Kubeflow Notebooks, Katib, and more.

For more details, see our comparison between Kubeflow and deployKF.

How can I get involved with deployKF?

The deployKF project is a welcoming community of contributors and users. We encourage participation from anyone who shares our mission of making it easy to build open ML Platforms on Kubernetes.

For more details, see our community page.

Getting Started

For full details on how to get started with deployKF, please see the getting started page.

Requirements

• the deploykf cli tool, found in the deployKF/cli repo
• a Kubernetes cluster (WARNING: we strongly recommend using a dedicated Kubernetes cluster for deployKF)
• ArgoCD is deployed on your cluster
• a private git repo (in which to store your generated manifests)

Step 1: Create values file

deployKF is configured using YAML files containing configs named "values" which behave similarly to those in Helm.

deployKF has a very large number of configurable values (more than 1500), but you can start by defining a few important ones, and then grow your values file over time.

We recommend you start by copying the sample-values.yaml file, which includes reasonable defaults that should work on any Kubernetes cluster.

The following values will always need to be changed to match your environment:

Value	Description
argocd.source.repo.url	the URL of your manifest git repo for example, if you are using a GitHub repo named deployKF/examples, you might set this value to "https://github.com/deployKF/examples" or "[e-mail]:deployKF/examples.git" TIP: if you are using a private repo, you will need to configure your ArgoCD with the appropriate credentials
argocd.source.repo.revision	the git revision which contains your generated manifests for example, if you are using the main branch of your repo, you might set this value to "main"
argocd.source.repo.path	the path within your repo where the generated manifests are stored for example, if you are using a folder named GENERATOR_OUTPUT at the root of your repo, you might set this value to "./GENERATOR_OUTPUT/"

We are actively working on detailed "production usage" guides, but for now, here are some other values you might want to change:

Value	Description
deploykf_core.deploykf_auth.dex.connectors	connect with an external identity provider (e.g. Microsoft AD, Okta, GitHub, Google, etc.)
deploykf_core.deploykf_auth.dex.staticPasswords	create user accounts for your team (if not using an external identity provider)
deploykf_core.deploykf_dashboard.navigation.externalLinks	add custom links to the dashboard navigation menu
deploykf_core.deploykf_istio_gateway	configure the istio ingress gateway (make it accessible from outside the cluster)
deploykf_core.deploykf_profiles_generator.profiles	create profiles (namespaces) and assign groups and users to them
kubeflow_tools.katib.mysql	configure an external MySQL database for Katib
kubeflow_tools.notebooks.spawnerFormDefaults	configure Kubeflow Notebooks, including notebook images, GPU resources, and more
kubeflow_tools.pipelines.mysql	configure an external MySQL database for Kubeflow Pipelines
kubeflow_tools.pipelines.objectStore	configure an external object store (like S3) for Kubeflow Pipelines

For information about other values, you can refer to the following resources:

• docstrings in generator/default_values.yaml, which contains defaults for all values
• the values reference page, which contains a list of all values with links to their docstrings
• the "topics" section of the website, which has information about achieving specific goals, such as using an external S3-compatible object store

TIP: for a refresher on YAML syntax, we recommend Learn YAML in Y minutes and YAML Multiline Strings

Step 2: Generate manifests

You must generate your manifests and commit them to a git repo before ArgoCD can deploy them to your cluster.

The generate command of the deploykf CLI creates a manifests folder for a specific version of deployKF and one or more values files:

deploykf generate \
    --source-version "0.1.0" \
    --values ./custom-values.yaml \
    --output-dir ./GENERATOR_OUTPUT

After running deploykf generate, you will likely want to commit the changes to your repo:

# for example, to directly commit changes to the 'main' branch of your repo
git add GENERATOR_OUTPUT
git commit -m "my commit message"
git push origin main

WARNING: any manual changes made in the --output-dir will be overwritten each time the deploykf generate command runs, so please only make changes in your --values files. If you find yourself needing to make manual changes, this indicates we might need a new value, so please raise an issue to help us improve the project!

TIP: the --source-version can be any valid deployKF version, see the changelog for a list of versions

TIP: if you specify --values multiple times, they will be merged with later ones taking precedence (note, YAML lists are not merged, they are replaced in full)

Step 3: Apply app-of-apps

The only manifest you need to manually apply is the ArgoCD app-of-apps, which creates all the other ArgoCD applications.

The app-of-apps.yaml manifest is generated at the root of your --output-dir folder, so you can apply it with:

kubectl apply --filename GENERATOR_OUTPUT/app-of-apps.yaml

Step 4: Access ArgoCD

If this is the first time you are using ArgoCD, you will need to retrieve the initial password for the admin user:

echo $(kubectl -n argocd get secret/argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

This kubectl command will port-forward the argocd-server Service to your local machine:

kubectl port-forward --namespace "argocd" svc/argocd-server 8090:https

You should now see the ArgoCD interface at https://localhost:8090, where you can log in with the admin user and the password you retrieved in the previous step.

Step 5: Sync applications

You can now sync the ArgoCD applications which make up deployKF.

Group Name	Group Label	ArgoCD Application Names
app-of-apps		deploykf-app-of-apps
deploykf-dependencies	app.kubernetes.io/component=deploykf-dependencies	dkf-dep--cert-manager, dkf-dep--istio, dkf-dep--kyverno
deploykf-core	app.kubernetes.io/component=deploykf-core	dkf-core--deploykf-auth, dkf-core--deploykf-dashboard, dkf-core--deploykf-istio-gateway, dkf-core--deploykf-profiles-generator
deploykf-opt	app.kubernetes.io/component=deploykf-opt	dkf-opt--deploykf-minio, dkf-opt--deploykf-mysql
deploykf-tools	app.kubernetes.io/component=deploykf-tools	N/A
kubeflow-dependencies	app.kubernetes.io/component=kubeflow-dependencies	kf-dep--argo-workflows
kubeflow-tools	app.kubernetes.io/component=kubeflow-tools	kf-tools--katib, kf-tools--notebooks--jupyter-web-app, kf-tools--notebooks--notebook-controller, kf-tools--pipelines, kf-tools--poddefaults-webhook, kf-tools--tensorboards--tensorboard-controller, kf-tools--tensorboards--tensorboards-web-app, kf-tools--training-operator, kf-tools--volumes--volumes-web-app

WARNING: you must sync each "group" of applications in the same order as the table to avoid dependency issues

WARNING: some applications, specifically dkf-dep--cert-manager and dkf-core--deploykf-profiles-generator may fail to sync on the first attempt, simply wait a few seconds and try the sync again

TIP: you may also sync the applications with the argocd CLI, but we recommend syncing with the web interface when you are first getting started so you can debug any issues:

# expose ArgoCD API server
kubectl port-forward svc/argocd-server -n argocd 8090:https

# get the admin password (if you have not changed it)
argocd "admin" initial-password -n argocd

# log in to ArgoCD
ARGOCD_PASSWORD="<YOUR_PASSWORD_HERE>"
argocd login localhost:8090 --username "admin" --password "$ARGOCD_PASSWORD" --insecure

# sync the apps
argocd app sync "deploykf-app-of-apps"
argocd app sync -l "app.kubernetes.io/component=deploykf-dependencies"
argocd app sync -l "app.kubernetes.io/component=deploykf-core"
argocd app sync -l "app.kubernetes.io/component=deploykf-opt"
argocd app sync -l "app.kubernetes.io/component=deploykf-tools"
argocd app sync -l "app.kubernetes.io/component=kubeflow-dependencies"
argocd app sync -l "app.kubernetes.io/component=kubeflow-tools"

Step 6: Access deployKF

If you have not configured a public Service for your deploykf-istio-gateway, you may access the deployKF web interface with kubectl port-forwarding.

First, you will need to add some lines to your /etc/hosts file (this is needed because Istio uses the "Host" header to route requests to the correct VirtualService).

For example, if you have set the deploykf_core.deploykf_istio_gateway.gateway.hostname value to "deploykf.example.com", you would add the following lines:

127.0.0.1 deploykf.example.com
127.0.0.1 argo-server.deploykf.example.com
127.0.0.1 minio-api.deploykf.example.com
127.0.0.1 minio-console.deploykf.example.com

This kubectl command will port-forward the deploykf-gateway Service to your local machine:

kubectl port-forward \
  --namespace "deploykf-istio-gateway" \
  svc/deploykf-gateway 8080:http 8443:https

You should now see the deployKF dashboard at https://deploykf.example.com:8443/, where you can use one of the following credentials (if you have not changed them):

User	Username	Password
Admin (Profile Owner)	[e-mail]	admin
User 1	[e-mail]	user1
User 2	[e-mail]	user2

Troubleshooting

ERROR: pods fail with "too many open files"

This error has been discussed in the upstream kubeflow repo, to resolve it you will need to increase your system's open/watched file limits.

On linux, you may need to increase the fs.inotify.max_user_* sysctl values, here are some values which users have reported to work:

1. Modify /etc/sysctl.conf to include the following lines:
   • fs.inotify.max_user_instances = 1280
   • fs.inotify.max_user_watches = 655360
2. Reload sysctl configs by running sudo sysctl -p