Support Service Account Keys defined in ImagePullSecrets
jnauska opened this issue · comments
What steps did you take and what happened:
We're using Google Artifact Registry to host our containers.
We have configured the access according to the Google Documentation: https://cloud.google.com/artifact-registry/docs/access-control
So we're using Service Account keys in our ImagePullSecrets. ImagePullSecrets would look like this (SA_KEY is encoded):
{
"auths": {
"repository_name": {
"username": "_json_key_base64",
"password": "SA_KEY"
}
}
}
Decoded SA_KEY would look like
{
"type": "service_account",
"project_id": "XYZ",
"private_key_id": "XYZ",
"private_key": "XYZ",
"client_email": "XYZ@XYZ",
"client_id": "XYZ",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "XYZ"
}
When Vulnerability Scan tries to Pull the image for scanning it gives out error:
"error":"reading .dockerconfigjson field of \"APPLICATION_NAME/image-pull-secrets\" secret: expected username and password concatenated with a colon (:)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227"}
starboard/pkg/docker/config.go
Lines 26 to 36 in 8944e54
Is splitting the string with :
and giving out errors if len(split) != 2
. As the SA_KEY has multiple :
included in the string, it gives out the error and breaks ImagePull
What did you expect to happen:
To be able to use Service Account keys in ImagePullSecrets.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
- Starboard version (use
starboard version
): 0.13.1 (starboard-operator) - Kubernetes version (use
kubectl version
): v1.20.10-gke.1600 - OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): N/A
How is this issue different from #279 ? In the current implementation Starboard reads imagePullSecrets to get username and password that are eventually passed to Trivy as TRIVY_USERNAME
and TRIVY_PASSWORD
environment variables. That's the only officially supported authentication scheme.
For GCE, ECR, and other managed registries more work as to be done.
I guess that purpose of the issue is the same, just that the description of the issue seemed that there was a working method, but there was support documentation needed.
And this issue is not specifically tied to GCR, but to other registries that are also using the same scheme for authentication.
I see what you mean now. The title of this issue confused me. So basically we're talking about a bug in parsing passwords from imagePullSecrets that may contain the colon (:
) characters. If we fix the parsing logic would that resolve your issue @jnauska ?
I tested with trivy (0.21.2) binary that it works:
export TRIVY_USERNAME="_json_key_base64"
export TRIVY_PASSWORD="$(cat /home/jnauska/Documents/gcp/registry_key | base64 -w0)"
jnauska@ububox:~/Downloads$ trivy --debug image REGION-docker.pkg.dev/PROJECT/FOLDER/IMAGE:IMAGETAG
2021-12-17T12:52:51.213+0200 DEBUG Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2021-12-17T12:52:51.215+0200 DEBUG cache dir: /home/jnauska/.cache/trivy
....
....
....
2021-12-17T12:52:54.777+0200 DEBUG Detecting library vulnerabilities, type: node-pkg, path:
REGION-docker.pkg.dev/PROJECT/FOLDER/IMAGE:IMAGETAG (debian 10.10)
================================================================================================================================
Total: 47 (UNKNOWN: 0, LOW: 24, MEDIUM: 12, HIGH: 9, CRITICAL: 2)
We got it working with the following ImagePullSecret setup:
{
"auths": {
"repository_name": {
"auth": "_json_key_base64:SA_KEY(encoded)"
}
}
}
So the issue is actually that starboard doesn't support username:password, but expects the username:password to be within auth.
So I guess this can be closed, though it would be nice to support username:password aswell w/o auth. And maybe add documentation that SA KEY's can be used with this kind of ImagePullSecret.
Our ImagePullSecret currently has both options:
{
"auths": {
"repository_name": {
"auth": "_json_key_base64:SA_KEY(encoded)"
"username": "_json_key_base64"
"password": "SA_KEY(encoded)"
}
}
}
We got it working with the following ImagePullSecret setup:
{ "auths": { "repository_name": { "auth": "_json_key_base64:SA_KEY(encoded)" } } }
So the issue is actually that starboard doesn't support username:password, but expects the username:password to be within auth.
So I guess this can be closed, though it would be nice to support username:password aswell w/o auth. And maybe add documentation that SA KEY's can be used with this kind of ImagePullSecret.
Our ImagePullSecret currently has both options:
{ "auths": { "repository_name": { "auth": "_json_key_base64:SA_KEY(encoded)" "username": "_json_key_base64" "password": "SA_KEY(encoded)" } } }
Interesting. We've discovered the same limitation this week and reported in #855 . To confirm my understanding, will #855 solve your problem? /cc @deven0t
Hey @jnauska,
Thank for sharing this workaround.
I tried using with with the starboard-operator version 0.14.1, however I keep on getting the following errors:
{"level":"error","ts":1643803264.6358004,"logger":"controller.replicaset","msg":"Reconciler error","reconciler group":"apps","reconciler kind":"ReplicaSet","name":"nginx-deployment-5c59b4886f","namespace":"web","error":"reading .dockerconfigjson field of \"web/gcr-registry\" secret: expected username and password concatenated with a colon (:)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227"}
The gcr-registry secret's content looks like this:
{
"auths": {
"gcr_repository_name_1": {
"auth": "_json_key_base64:SA_KEY(b64 encoded)",
"username": "_json_key_base64(cleartext)",
"password": "SA_KEY(b64 encoded)"
},
"gcr_repository_name_2": {
"auth": "_json_key_base64:SA_KEY(b64 encoded)",
"username": "_json_key_base64(cleartext)",
"password": "SA_KEY(b64 encoded)"
}
}
}
Any suggestions?
Remove the username
and password
entries from the auths, and just rely on the auth
Hey @jnauska,
Thank you for your response.
By default the gcr-registry secret only has the auth parameter set and it looks like this in practise:
{
"auths": {
"gcr_repository_name_1": {
"auth": "_json_key:SA_KEY(b64 encoded)"
},
"gcr_repository_name_2": {
"auth": "_json_key:SA_KEY(b64 encoded)"
}
}
}
I added the username & password as a test, with the secret shared above I still get the same errors.
I think you need to double encode that, so like:
"auth": "base64(_json_key_base64:base64(SA_KEY))"
Hello @jnauska,
Thank you for your response.
I did a few tests to get closer to the solution:
base64:(_json_key:SA_KEY)
- Image pull successful, "Reconciler error" messages in the Operatorcleartext:(_json_key:SA_KEY)
- Image pull successful, "Reconciler error" messages in the Operatorbase64:(_json_key_base64:base64(SA_KEY))
- Image pull fails, Operator gives the following error:
{"level":"error","ts":1643975080.345095,"logger":"reconciler.vulnerabilityreport","msg":"Scan job container","job":"security/scan-vulnerabilityreport-75d5f7b9d5","container":"alpine","status.reason":"Error","status.message":"2022-02-04T11:44:39.689Z\t\u001b[31mFATAL\u001b[0m\tscan error: unable to initialize a scanner: unable to initialize a docker scanner: 3 errors occurred:\n\t* unable to inspect the image (XY.gcr.io/<PATH>/alpine:3.15.0): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* GET https://XY.gcr.io/v2/token?scope=repository%3A<project_name>%2F<repo_name>%2F<repo_dir>%3Apull&service=XY.gcr.io: UNAUTHORIZED: Not Authorized.\n\n\n","stacktrace":"github.com/aquasecurity/starboard/pkg/operator/controller.(*VulnerabilityReportReconciler).reconcileJobs.func1\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/vulnerabilityreport.go:320\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
The scan job secrets have the following content (when testing option 3):
alpine.password
: base64:SA_KEY
alpine.username
: _json_key_base64
Note: The image pull secret contains auths.repository_name.auth
field only (without username or password specified)
@jnauska & @danielpacak,
Do you have any suggestions please?
Hello @jessequinn,
None of the above mentioned workarounds/solutions worked for me unfortunately.
The following looks like HOW it should work. Basically someone would need to update the plugin to place an empty username and add a new ENV VAR GOOGLE_APPLICATION_CREDENTIALS. I may try to make a PR for this. I tested the docker example given in that PR. It works.
ok. i have played with the code. Actually i think the problem could be fixed quite easily.
docker/config.go
contains the following:
func decodeAuths(auths map[string]Auth) (map[string]Auth, error) {
decodedAuths := make(map[string]Auth)
for server, entry := range auths {
if entry == (Auth{}) {
continue
}
if strings.TrimSpace(string(entry.Auth)) == "" {
decodedAuths[server] = Auth{
Username: entry.Username,
Password: entry.Password,
}
continue
}
// TODO: issue decoding GCR auth
username, password, err := entry.Auth.Decode()
if err != nil {
return nil, err
}
decodedAuths[server] = Auth{
Auth: entry.Auth,
Username: username,
Password: password,
}
}
return decodedAuths, nil
}
now the Decode() method using SplitN rather than Split removes the issue:
func (v *BasicAuth) Decode() (string, string, error) {
bytes, err := base64.StdEncoding.DecodeString(string(*v))
if err != nil {
return "", "", err
}
split := strings.SplitN(string(bytes), ":", 2)
fmt.Println("split", len(split), split[0], split[1])
if len(split) != 2 {
return "", "", fmt.Errorf("expected username and password concatenated with a colon (:)")
}
return split[0], split[1], nil
}
I tried testing this through Kind as per the contribution guidelines; however, modifications to the config.go DO NOT APPEAR TO APPLY to the docker images built. Any idea? @danielpacak
Added a PR for the SplitN #1126