This repository serves as an example for managing multi-tenant Kafka clusters with Git and Flux v2.
The Confluent Openshift example repository uses the Flux CLI to create the necessary manifests; making it super confusing to follow. To help visualize the workflow, we will be working with the following Repositories with arrows representing the Flux Kustomization resources linking things together.
- Complete installation of demo cluster
- Flux CLI
- k9s (optional but good idea)
The Confluent Openshift example repository contains the following directory structure under the cluster-manifests:
- clusters dir contains the Flux configuration per cluster
- operators dir contains common operators such as Confluent, CRDs and cluster-wide polices
├── cluster-manifests │ ├── clusters │ │ ├── development-ocp │ │ │ └── flux-system │ │ │ ├── gotk-components.yaml │ │ │ ├── gotk-sync.yaml │ │ │ ├── kustomization.yaml │ │ │ └── operators.yaml │ │ └── production-ocp │ │ └── flux-system │ │ ├── gotk-components.yaml │ │ ├── gotk-sync.yaml │ │ ├── kustomization.yaml │ │ └── operators.yaml │ └── operators │ ├── confluent.yaml │ ├── kustomization.yaml │ └── namespaces.yaml └── policy └── confluent-security-context.yaml
A tenant repository contains the following top directories for deploying Kafka in their own isolated namespace:
- confluent dir contains Confluent component CRD manifests. Topology of the Kafka cluster you want to deploy
- connectors dir contains Kafka connectors with configuration / credentials in Confluent CRD format
- topics dir contains all topics for current tenant in Confluent CRD format
├── confluent │ ├── ca-pair-sslcerts.yaml │ ├── connect-cluster.yaml │ ├── kafka.yaml │ ├── kustomization.yaml │ ├── schema-registry.yaml │ ├── tls-group1.yaml │ └── zookeeper.yaml ├── connectors │ ├── kustomization.yaml │ └── sample-connector.yaml ├── kustomization.yaml └── topics ├── foobar.yaml ├── gitops4lyfe.yaml └── kustomization.yaml
Providing you have completed the steps of bootstrapping the cluster and have successfully deployed the Confluent operator you can now deploy any number of tenant isolated Kafka clusters. The Flux CLI offers commands to generate the Kubernetes manifests needed to define tenants with their RBAC permissions. Assuming the central Kafka admin wants to create a tenant named alpha-team with their apps having the ability to access Kafka in the same namespace:
- Fork this repository and create the sync manifests for the tenant Git repository:
flux create source git team-alpha-resources \ --namespace=team-alpha \ --url=https://github.com/<org>/<dev-team> \ --branch=main \ --export > ./tenants/base/team-alpha/sync.yaml flux create kustomization team-alpha-resources \ --namespace=team-alpha \ --service-account=team-alpha \ --source=GitRepository/team-alpha-resources \ --path="./" \ --export >> ./tenants/base/team-alpha/sync.yaml
Check out these related projects.
- Confluent for Kubernetes (CFK) examples - Playground for Kafka / Confluent Kubernetes experimentations
- Confluent Platform on Azure - Terraform module to deploy CP using supported Ansible scripts
- Confluent Platform on AWS - Terraform module to deploy CP using supported Ansible scripts
- CP Ansible Docker - Docker wrapper to run CP Ansible on air gapped environments
File a GitHub issue, send us an email or tweet us.
Copyright © 2017-2021 OSO | See LICENCE for full details.
We at OSO help teams to adopt emerging technologies and solutions to boost their competitiveness, operational excellence and introduce meaningful innovations that drive real business growth. Our developer-first culture, combined with our cross-industry experience and battle-tested delivery methods allow us to implement the most impactful solutions for your business.
Looking for support applying emerging technologies in your business? We’d love to hear from you, get in touch by email
Start adopting new technologies by checking out our other projects, follow us on twitter, join our team of leaders and challengers, or contact us to find the right technology to support your business.