Giters

enix / kube-image-keeper

kuik is a container image caching system for Kubernetes

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

registry needs a configured S3 endpoint to enable HA mode

felipewnp opened this issue · comments

commented

Hi.

I'm having the following problem when installing kuik with ReadWriteMany and replica > 1

Error: execution error at (kube-image-keeper/templates/registry-statefulset.yaml:4:3): registry needs a configured S3 endpoint to enable HA mode (>1 replicas), please enable minio or configure an external S3 endpoint

This is my kuik helm values.yaml file (the registry part):

registry:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - registry
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - kube-image-keeper
          topologyKey: failure-domain.beta.kubernetes.io/zone
  replicas: 2
  nodeSelector:
    intent: system
  pdb:
    create: true
    minAvailable: 1
  persistence:
    enabled: true
    accessModes: "ReadWriteMany"
    size: 1Gi # EFS doesn't have size parameter
    storageClass: efs
  priorityClassName: system-cluster-critical

Have I done something wrong? Or are the docs outdated?

At this link, It says that kuik HA works with ReadWriteMany and EFS.

Also, I'm only using EFS because I'm afraid of transfer costs from S3 to kuik. Does that (transfer) usually happen a lot?

commented

Hi @felipewnp, I'm not the maintainer of this project -- but I did contribute the feature you are talking about :) Are you using the latest version, v1.7.0-beta.2?

The only reason I'm asking is because the old error message used to say the exception you are getting from Helm, whereas this feature was just added into the above version, and if there was a problem would say:

registry needs a configured S3 endpoint or a PVC which supports ReadWriteMany to enable HA mode (>1 replicas), please enable minio or configure an external S3 endpoint

commented

Hi @felipewnp , as @thomasjcf29 said, the feature is available in the latest beta version.
Which version are you using ?

commented

Hi @donch @thomasjcf29

[...] the feature is available in the latest beta version [...]

Yep, I missed that. Found in the release notes:

registry-statefulset: added an additionally supported HA mode (50a24dc)

Should I open a PR to mention in the high availability guide that the PVC ReadWriteMany mode is only available in the v1.7.0-beta.2 and above?

I was using the latest v1.6.0 release and will try the v1.7.0-beta.2.





Side note:

Have anyone used S3 persistence with IRSA?

I tried to do that (without setting access_key_id and access_key_secret) but looks like registry doesn't work with IRSA until 3.0.0-alpha.1.

As the issue says, the image registry:3.0.0-alpha.1 indeed works with IRSA but the registry responds with HTTP 404 or HTTP 5XX when a node tries to pull any image, not showing any particularly useful error message in the logs, besides the HTTP 4XX / 5XX codes..

commented

Should I open a PR to mention in the high availability guide that the PVC ReadWriteMany mode is only available in the v1.7.0-beta.2 and above?

We are discussing internally to found a better way to display this kind of information.

Side note:

Have anyone used S3 persistence with IRSA?

Not yet :)

commented

I close this issue, feel free to reopen if needed