permissions grant with IAM Roles for Service Accounts does not work

[Sabri Sayilgan]

Hi There,

I am leveraging chartmuseum --version 3.8.0. As I am using service account with proper iam role on aws, I can not bring up Chartmuseum pod. Application does not throw any error even though --debug is enabled. If use worker node's role and attach necessary policies it works properly. I can make it work with iam user's credentials(aws_access_key_id and aws_secret_access_key).

Reproduce:

cat << custom-config.yaml > EOF
env:
open:
# STORAGE: local
AWS_SDK_LOAD_CONFIG: true
STORAGE: amazon
STORAGE_AMAZON_BUCKET:
STORAGE_AMAZON_PREFIX:
STORAGE_AMAZON_REGION: us-west-1
DEBUG: true
DISABLE_API: false
AUTH_ANONYMOUS_GET: true
ALLOW_OVERWRITE: true

service:
type: NodePort
serviceAccount:
create: true
automountServiceAccountToken: true
annotations:
eks.amazonaws.com/role-arn: ""

image:
repository: < my image reposiyory>
tag: v0.14.0
pullPolicy: IfNotPresent
securityContext:
enabled: true
fsGroup: 65534
EOF

helm install test -f custom-config chartmuseum/chartmuseum --version 3.8.0

After running this command, pod is trying to come up for a while. Since readiness and liveness probes are failing, pod is crashing eventually. When describe I am getting something like shared below:

Readiness probe failed: Get "http://<POD_IP>:8080/health": dial tcp <POD_IP>:8080: connect: connection refused

Note: When the same service account attached to an aws cli pod, it can list s3 bucket objets with no issue.

Any idea why it is behaving this way?

[Casey Buto]

👋 @sabrisay there should be some logs available from the container, those would help with debugging. At first glance, it looks like STORAGE_AMAZON_BUCKET: isn't set, can you try setting the bucket name?

[Yasser Alattas]

I am experiencing access denied to the bucket as ServiceAccount is not created via HelmCharts even though its specified in the values.yaml

env:
  open:
    DEBUG: true
    DISABLE_METRICS: false
    CHART_URL: https://charts.domain.com
    STORAGE: amazon
    STORAGE_AMAZON_BUCKET: BUCKET_NAME
    STORAGE_AMAZON_PREFIX: PREFEIX
    STORAGE_AMAZON_REGION: us-east-1
  service:
    servicename: chartmuseum
  serviceMonitor:
    enabled: true
  serviceAccount:
    create: true
    name: chartmuseum
    annotations:
      iam.amazonaws.com/role: ROLE_NAME

I am wondering why ServiceAccount isn't created. Even helm diff showing everything. Yet, serviceAccount is default

repositories:
  - name: chartmuseum
    url: https://chartmuseum.github.io/charts

Am I pointing to the deprecated repo ? if so, what the updated one. I've been looking for a valid values.yaml

CORRECTION

I found the correct values and I adjusted my values accordingly

env:
  open:
    DEBUG: true
    DISABLE_METRICS: false
    CHART_URL: https://charts.domain.com
    # storage backend, can be one of: local, alibaba, amazon, google, microsoft, oracle
    STORAGE: amazon
    STORAGE_AMAZON_BUCKET: BUCKET_NAME
    STORAGE_AMAZON_PREFIX: charts
    STORAGE_AMAZON_REGION: us-east-1
service:
  servicename: chartmuseum
serviceMonitor:
  enabled: true
serviceAccount:
  create: true
  name: chartmuseum-s3
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::12345678912:role/chartmuseum-s3

ServiceAccount got created and attached to the pod. Yet, I am getting AccessDenied on container logs but not sure coming from where

role trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::12345678912:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/OIDC_ID"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-central-1.amazonaws.com/id/OIDC_ID:sub": "system:serviceaccount:chartmuseum:chartmuseum-s3"
                }
            }
        }
    ]
}

Indeed my cluster lives in eu-central-1 and bucket in us-east-1 so not a typo :)

IAM policy attached to the role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        },
        {
            "Action": [
                "s3:ListBucketVersions",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucket",
                "s3:GetEncryptionConfiguration",
                "s3:GetBucketVersioning"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::BUCKET_NAME"
        }
    ]
}