jomeier / operator

MinIO Operator for kubernetes (k8s) https://kubernetes.io/

MinIO Operator

MinIO is a Kubernetes-native high performance object store with an S3-compatible API. The MinIO Kubernetes Operator supports deploying MinIO Tenants onto private and public cloud infrastructures ("Hybrid" Cloud).

Table of Contents

• Architecture
  • MinIO Console
  • MinIO Operator and kubectl Plugin
• Create a MinIO Tenant
• Expand a MinIO Tenant
• Kubernetes Cluster Configuration
  • Default Storage Class
  • Local Persistent Volumes
  • MinIO Tenant Namespace

Architecture

Each MinIO Tenant represents an independent MinIO Object Store within the Kubernetes cluster. The following diagram describes the architecture of a MinIO Tenant deployed into Kubernetes:

IMAGE

MinIO provides multiple methods for accessing and managing the MinIO Tenant:

MinIO Console

The MinIO Console provides a graphical user interface (GUI) for interacting with MinIO Tenants.

IMAGE

Administrators of MinIO Tenants can perform a variety of tasks through the Console, including user creation, policy configuration, and bucket replication. The Console also provides a high level view of Tenant health, usage, and healing status.

For more complete documentation on using the MinIO Console, see the MinIO Console Github Repository.

MinIO Operator and kubectl Plugin

The MinIO Operator extends the Kubernetes API to support deploying MinIO-specific resources as a Tenant in a Kubernetes cluster.

The MinIO kubectl minio plugin wraps the Operator to provide a simplified interface for deploying and managing MinIO Tenants in a Kubernetes cluster through the kubectl command line tool.

Create a MinIO Tenant

This procedure creates a 4-node MinIO Tenant suitable for evaluation and early development using MinIO for object storage.

Prerequisites

• MinIO requires Kubernetes version 1.17.0 or later.

• This procedure assumes the cluster contains a namespace for the MinIO Tenant.

• This procedure assumes the cluster contains a StorageClass for the MinIO Tenant Persistent Volumes (PV). The StorageClass must have volumeBindingMode: WaitForFirstConsumer

• This procedure uses the Kubernetes krew plugin manager. See the krew installation documentation.

1) Install the MinIO Operator

Install using kubectl krew

Run the following command to install the MinIO Operator and Plugin using krew:

   kubectl krew update
   kubectl krew install minio

Run the following command to initialize the Operator:

kubectl minio init

2) Create a New Tenant

The following kubectl minio command creates a MinIO Tenant with 4 nodes, 16 volumes, and a total capacity of 16Ti. This configuration requires at least 16 Persistent Volumes.

   kubectl minio tenant create minio-tenant-1 \
      --servers 4                             \
      --volumes 16                            \
      --capacity 16Ti                         \
      --namespace minio-tenant-1              \
      --storageClassName local-storage        \

• The minio-tenant-1 argument specifies the name of the MinIO Tenant. The MinIO Operator uses this name as a prefix for certain resources in the Tenant.

• The --servers field indicates the number of minio pods to deploy into the cluster. The cluster must have at least one available worker Node per minio pod.

• The --volumes field indicates the total number of volumes in the Tenant. MinIO generates a Persistent Volume Claim (PVC) for each volume and evenly distributes volumes across each minio pod. The example above results in 4 volumes per minio pod.

  Tenant creation hangs if the Kubernetes cluster does not have at least one unbound Persistent Volume (PV) for each generated PVC.

• The --capacity field indicates the total capacity of the cluster. MinIO determines the amount of storage to request for each pvc by dividing the specified capacity by the total number of volumes in the server. The example above results in 1Ti requested capacity per volume.

  Tenant creation hangs if the Kubernetes cluster does not have at least one Persistent Volume (PV) with sufficient capacity to bind to each generated PVC.

• The --namespace field indicates the namespace onto which MinIO deploys the Tenant. If omitted, MinIO uses the Default namespace.

  MinIO supports one MinIO Tenant per namespace.

• The --storageClassName field indicates which StorageClass to use when generating each PVC.

3) Connect to the Tenant

MinIO outputs credentials for connecting to the MinIO Tenant as part of the creation process:

Tenant 'minio-tenant-1' created in 'minio-tenant-1' Namespace
  Username: admin 
  Password: dbc978c2-bfbe-41bf-9dc6-699c76bafcd0 
+-------------+------------------------+------------------+--------------+-----------------+
| APPLICATION |      SERVICE NAME      |     NAMESPACE    | SERVICE TYPE | SERVICE PORT(S) |
+-------------+------------------------+------------------+--------------+-----------------+
| MinIO       | minio                  | minio-tenant-1   | ClusterIP    | 443             |
| Console     | minio-tenant-1-console | minio-tenant-1   | ClusterIP    | 9090,9443       |
+-------------+------------------------+------------------+--------------+-----------------+

Copy the credentials to a secure location, such as a password protected key manager. MinIO does not display these credentials again.

MinIO Tenants deploy with TLS enabled by default, where the MinIO Operator uses the Kubernetes certificates.k8s.io API to generate the required x.509 certificates. Each certificate is signed using the Kubernetes Certificate Authority (CA) configured during cluster deployment. While Kubernetes mounts this CA on Pods in the cluster, Pods do not trust that CA by default. You must copy the CA to a directory such that the update-ca-certificates utility can find and add it to the system trust store to enable validation of MinIO TLS certificates:

cp /var/run/secrets/kubernetes.io/serviceaccount/ca.crt /usr/local/share/ca-certificates/
update-ca-certificates

For applications external to the Kubernetes cluster, you must configure Ingress or a Load Balancer to expose the MinIO Tenant services. Alternatively, you can use the kubectl port-forward command to temporarily forward traffic from the local host to the MinIO Tenant.

• The minio service provides access to MinIO Object Storage operations.

• The minio-tenant-1-console service provides access to the MinIO Console. The MinIO Console supports GUI administration of the MinIO Tenant.

Expand a MinIO Tenant

MinIO supports expanding an existing MinIO Tenant onto additional hosts and storage.

• MinIO requires Kubernetes version 1.17.0 or later.

• This procedure assumes the cluster contains a namespace for the MinIO Tenant.

The following kubectl minio command expands a MinIO Tenant with an additional 4 minio pods, 16 volumes, and added capacity of 16Ti:

   kubectl minio tenant expand minio-tenant-1 \
      --servers 4                             \
      --volumes 16                            \
      --capacity 16Ti

• The minio-tenant-1 argument specifies the name of the existing MinIO Tenant to expand.

• The --servers field indicates the number of minio pods to deploy into the cluster. The cluster must have at least one available worker Node per minio pod.

• The --volumes field indicates the total number of volumes in the Tenant. MinIO generates a Persistent Volume Claim (PVC) for each volume and evenly distributes volumes across each minio pod. The example above results in 4 volumes per minio pod.

  Tenant expansion hangs if the Kubernetes cluster does not have at least one unbound Persistent Volume (PV) for each generated PVC.

• The --capacity field indicates the total capacity of the cluster. MinIO determines the amount of storage to request for each pvc by dividing the specified capacity by the total number of volumes in the server. The example above results in 1Ti requested capacity per volume.

  Tenant expansion hangs if the Kubernetes cluster does not have at least one unbound Persistent Volume (PV) for each generated PVC.

Kubernetes Cluster Configuration

Default Storage Class

The MinIO Kubernetes Plugin (kubectl minio) automatically generates Persistent Volume Claims (PVC) as part of deploying a MinIO Tenant. The plugin defaults to creating each PVC with the default Kubernetes Storage Class.

MinIO Tenants require that the StorageClass set volumeBindingMode to WaitForFirstConsumer. The default StorageClass may use the Immediate setting, which can cause complications during PVC binding. MinIO strongly recommends creating a custom StorageClass for use by PV supporting a MinIO Tenant:

The following StorageClass object contains the appropriate fields for use with the MinIO Plugin:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
    name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

To specify the storage class, include the --storageClassName option to kubectl minio tenant create.

Local Persistent Volumes

MinIO automatically creates Persistent Volume Claims (PVC) as part of Tenant creation. Ensure the cluster has at least one Persistent Volume for each PVC MinIO requests.

You can estimate the number of PVC by multiplying the number of minio server pods in the Tenant by the number of drives per node. For example, a 4-node Tenant with 4 drives per node requires 16 PVC and therefore 16 PV.

MinIO strongly recommends using the following CSI drivers for creating local PV to ensure best object storage performance:

• Local Persistent Volume
• OpenEBS Local PV

MinIO Tenant Namespace

MinIO supports no more than one MinIO Tenant per Namespace. The following kubectl command creates a new namespace for the MinIO Tenant.

kubectl create namespace minio-tenant1

License

Use of MinIO Operator is governed by the GNU AGPLv3 or later, found in the LICENSE file.

Explore Further

• Create a MinIO Tenant.
• TLS for MinIO Tenant.
• Examples for MinIO Tenant Settings
• Custom Hostname Discovery.
• Apply PodSecurityPolicy.
• Deploy MinIO Tenant with Console.
• Deploy MinIO Tenant with KES.