Giters

aquasecurity / starboard

Moved to https://github.com/aquasecurity/trivy-operator

Home Page:https://aquasecurity.github.io/starboard/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support Service Account Keys defined in ImagePullSecrets

jnauska opened this issue · comments

commented

What steps did you take and what happened:

We're using Google Artifact Registry to host our containers.

We have configured the access according to the Google Documentation: https://cloud.google.com/artifact-registry/docs/access-control

So we're using Service Account keys in our ImagePullSecrets. ImagePullSecrets would look like this (SA_KEY is encoded):

{
  "auths": {
    "repository_name": {
      "username": "_json_key_base64",
      "password": "SA_KEY"
    }
  }
}

Decoded SA_KEY would look like

{
  "type": "service_account",
  "project_id": "XYZ",
  "private_key_id": "XYZ",
  "private_key": "XYZ",
  "client_email": "XYZ@XYZ",
  "client_id": "XYZ",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "XYZ"
}

When Vulnerability Scan tries to Pull the image for scanning it gives out error:

"error":"reading .dockerconfigjson field of \"APPLICATION_NAME/image-pull-secrets\" secret: expected username and password concatenated with a colon (:)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227"}

starboard/pkg/docker/config.go

Lines 26 to 36 in 8944e54

func (v *BasicAuth) Decode() (string, string, error) {
bytes, err := base64.StdEncoding.DecodeString(string(*v))
if err != nil {
return "", "", err
}
split := strings.Split(string(bytes), ":")
if len(split) != 2 {
return "", "", fmt.Errorf("expected username and password concatenated with a colon (:)")
}
return split[0], split[1], nil
}

Is splitting the string with : and giving out errors if len(split) != 2. As the SA_KEY has multiple : included in the string, it gives out the error and breaks ImagePull

What did you expect to happen:

To be able to use Service Account keys in ImagePullSecrets.

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Starboard version (use starboard version): 0.13.1 (starboard-operator)
  • Kubernetes version (use kubectl version): v1.20.10-gke.1600
  • OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): N/A
commented

How is this issue different from #279 ? In the current implementation Starboard reads imagePullSecrets to get username and password that are eventually passed to Trivy as TRIVY_USERNAME and TRIVY_PASSWORD environment variables. That's the only officially supported authentication scheme.

For GCE, ECR, and other managed registries more work as to be done.

commented

I guess that purpose of the issue is the same, just that the description of the issue seemed that there was a working method, but there was support documentation needed.

And this issue is not specifically tied to GCR, but to other registries that are also using the same scheme for authentication.

commented

I see what you mean now. The title of this issue confused me. So basically we're talking about a bug in parsing passwords from imagePullSecrets that may contain the colon (:) characters. If we fix the parsing logic would that resolve your issue @jnauska ?

commented

I tested with trivy (0.21.2) binary that it works:

export TRIVY_USERNAME="_json_key_base64"
export TRIVY_PASSWORD="$(cat /home/jnauska/Documents/gcp/registry_key | base64 -w0)"

jnauska@ububox:~/Downloads$ trivy --debug image REGION-docker.pkg.dev/PROJECT/FOLDER/IMAGE:IMAGETAG
2021-12-17T12:52:51.213+0200	DEBUG	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2021-12-17T12:52:51.215+0200	DEBUG	cache dir:  /home/jnauska/.cache/trivy
....
....
....
2021-12-17T12:52:54.777+0200	DEBUG	Detecting library vulnerabilities, type: node-pkg, path: 

REGION-docker.pkg.dev/PROJECT/FOLDER/IMAGE:IMAGETAG (debian 10.10)
================================================================================================================================
Total: 47 (UNKNOWN: 0, LOW: 24, MEDIUM: 12, HIGH: 9, CRITICAL: 2)
commented

We got it working with the following ImagePullSecret setup:

{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
    }
  }
}

So the issue is actually that starboard doesn't support username:password, but expects the username:password to be within auth.

So I guess this can be closed, though it would be nice to support username:password aswell w/o auth. And maybe add documentation that SA KEY's can be used with this kind of ImagePullSecret.

Our ImagePullSecret currently has both options:

{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
      "username": "_json_key_base64"
      "password": "SA_KEY(encoded)"
    }
  }
}
commented

We got it working with the following ImagePullSecret setup:

{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
    }
  }
}

So the issue is actually that starboard doesn't support username:password, but expects the username:password to be within auth.

So I guess this can be closed, though it would be nice to support username:password aswell w/o auth. And maybe add documentation that SA KEY's can be used with this kind of ImagePullSecret.

Our ImagePullSecret currently has both options:

{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
      "username": "_json_key_base64"
      "password": "SA_KEY(encoded)"
    }
  }
}

Interesting. We've discovered the same limitation this week and reported in #855 . To confirm my understanding, will #855 solve your problem? /cc @deven0t

commented

Hey @jnauska,
Thank for sharing this workaround.
I tried using with with the starboard-operator version 0.14.1, however I keep on getting the following errors:
{"level":"error","ts":1643803264.6358004,"logger":"controller.replicaset","msg":"Reconciler error","reconciler group":"apps","reconciler kind":"ReplicaSet","name":"nginx-deployment-5c59b4886f","namespace":"web","error":"reading .dockerconfigjson field of \"web/gcr-registry\" secret: expected username and password concatenated with a colon (:)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227"}

The gcr-registry secret's content looks like this:

{
  "auths": {
    "gcr_repository_name_1": {
      "auth": "_json_key_base64:SA_KEY(b64 encoded)",
      "username": "_json_key_base64(cleartext)",
      "password": "SA_KEY(b64 encoded)"
    },
    "gcr_repository_name_2": {
      "auth": "_json_key_base64:SA_KEY(b64 encoded)",
      "username": "_json_key_base64(cleartext)",
      "password": "SA_KEY(b64 encoded)"
    }
  }
}

Any suggestions?

commented

Remove the username and password entries from the auths, and just rely on the auth

commented

Hey @jnauska,
Thank you for your response.
By default the gcr-registry secret only has the auth parameter set and it looks like this in practise:

{
  "auths": {
    "gcr_repository_name_1": {
      "auth": "_json_key:SA_KEY(b64 encoded)"
    },
    "gcr_repository_name_2": {
      "auth": "_json_key:SA_KEY(b64 encoded)"
    }
  }
}

I added the username & password as a test, with the secret shared above I still get the same errors.

commented

I think you need to double encode that, so like:

"auth": "base64(_json_key_base64:base64(SA_KEY))"

commented

Hello @jnauska,
Thank you for your response.
I did a few tests to get closer to the solution:

  1. base64:(_json_key:SA_KEY) - Image pull successful, "Reconciler error" messages in the Operator
  2. cleartext:(_json_key:SA_KEY) - Image pull successful, "Reconciler error" messages in the Operator
  3. base64:(_json_key_base64:base64(SA_KEY)) - Image pull fails, Operator gives the following error:

{"level":"error","ts":1643975080.345095,"logger":"reconciler.vulnerabilityreport","msg":"Scan job container","job":"security/scan-vulnerabilityreport-75d5f7b9d5","container":"alpine","status.reason":"Error","status.message":"2022-02-04T11:44:39.689Z\t\u001b[31mFATAL\u001b[0m\tscan error: unable to initialize a scanner: unable to initialize a docker scanner: 3 errors occurred:\n\t* unable to inspect the image (XY.gcr.io/<PATH>/alpine:3.15.0): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* GET https://XY.gcr.io/v2/token?scope=repository%3A<project_name>%2F<repo_name>%2F<repo_dir>%3Apull&service=XY.gcr.io: UNAUTHORIZED: Not Authorized.\n\n\n","stacktrace":"github.com/aquasecurity/starboard/pkg/operator/controller.(*VulnerabilityReportReconciler).reconcileJobs.func1\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/vulnerabilityreport.go:320\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}

The scan job secrets have the following content (when testing option 3):
alpine.password: base64:SA_KEY
alpine.username: _json_key_base64

Note: The image pull secret contains auths.repository_name.auth field only (without username or password specified)

commented

@jnauska & @danielpacak,
Do you have any suggestions please?

commented

@dgdevops so based on your comment the solution provided by @jnauska does not work? I tried your 3rd option and also image pull failed for me.

commented

Hello @jessequinn,
None of the above mentioned workarounds/solutions worked for me unfortunately.

commented

The following looks like HOW it should work. Basically someone would need to update the plugin to place an empty username and add a new ENV VAR GOOGLE_APPLICATION_CREDENTIALS. I may try to make a PR for this. I tested the docker example given in that PR. It works.

commented

ok. i have played with the code. Actually i think the problem could be fixed quite easily.

docker/config.go contains the following:

func decodeAuths(auths map[string]Auth) (map[string]Auth, error) {
	decodedAuths := make(map[string]Auth)
	for server, entry := range auths {
		if entry == (Auth{}) {
			continue
		}

		if strings.TrimSpace(string(entry.Auth)) == "" {
			decodedAuths[server] = Auth{
				Username: entry.Username,
				Password: entry.Password,
			}
			continue
		}

		// TODO: issue decoding GCR auth
		username, password, err := entry.Auth.Decode()
		if err != nil {
			return nil, err
		}

		decodedAuths[server] = Auth{
			Auth:     entry.Auth,
			Username: username,
			Password: password,
		}

	}
	return decodedAuths, nil
}

now the Decode() method using SplitN rather than Split removes the issue:

func (v *BasicAuth) Decode() (string, string, error) {
	bytes, err := base64.StdEncoding.DecodeString(string(*v))
	if err != nil {
		return "", "", err
	}
	split := strings.SplitN(string(bytes), ":", 2)
	fmt.Println("split", len(split), split[0], split[1])
	if len(split) != 2 {
		return "", "", fmt.Errorf("expected username and password concatenated with a colon (:)")
	}
	return split[0], split[1], nil
}

I tried testing this through Kind as per the contribution guidelines; however, modifications to the config.go DO NOT APPEAR TO APPLY to the docker images built. Any idea? @danielpacak

Added a PR for the SplitN #1126