"Reconciler error" when scanning docker image from GCR private repository

[dgdevops]

What steps did you take and what happened:

• Deployed the starboard-operator to a GKE cluster & configured it to monitor all namespaces
• Deployed applications that use imagePullSecrets (gcr-registry) and image from GCR private repository
• Updated the gcr-registry secret with username & password parameters
• Checked the Operator logs and found dozens of "Reconciler error" messages:

{"level":"error","ts":1643803264.6358004,"logger":"controller.replicaset","msg":"Reconciler error","reconciler group":"apps","reconciler kind":"ReplicaSet","name":"nginx-deployment-5c59b4886f","namespace":"web","error":"reading .dockerconfigjson field of \"web/gcr-registry\" secret: expected username and password concatenated with a colon (:)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227"}

What did you expect to happen:
I expected the Operator to parse the username & passwords correctly and schedule the jobs for GCR images as well

Anything else you would like to add:
The gcr-registry secret's content looks like this:

{
  "auths": {
    "gcr_repository_name_1": {
      "auth": "_json_key_base64:SA_KEY(b64 encoded)",
      "username": "_json_key_base64(cleartext)",
      "password": "SA_KEY(b64 encoded)"
    },
    "gcr_repository_name_2": {
      "auth": "_json_key_base64:SA_KEY(b64 encoded)",
      "username": "_json_key_base64(cleartext)",
      "password": "SA_KEY(b64 encoded)"
    }
  }
}

Environment:

• Starboard version (use starboard version): 0.14.1 (GKE deployment)
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-gke.1300", GitCommit:"9352724e36d9c7c35609d30b2696b0de61d23aa0", GitTreeState:"clean", BuildDate:"2021-10-06T09:26:56Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}

• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Windows 10
• Kubernetes CRDs: LINK

Additional notes:
I checked #857 and configured the gcr-registry secret according to this

[dgdevops]

Hi,

I have tried to use Workload Identity to assign IAM permissions to the starboard-operator Kubernetes Service Account to access the private GCR in the similar fashion that was described for ECR access. As I checked the scan job's pods have the serviceAccountName specified however the automountServiceAccountToken part is set to false which is blocking it's use, can it be overwritten somehow to true?

serviceAccountName: starboard-operator
serviceAccount: starboard-operator
automountServiceAccountToken: false

Thanks,

[dgdevops]

Any suggestions here @danielpacak?

[dgdevops]

Hi,

It seems that Workload Identity also works with the automountServiceAccountToken set to false so there is no need to change that. However the operator still reports the "expected username and password concatenated with a colon" errors.
Is there any way to fix that? I will add the GCR integration how-to to the documentations once this gets clear.

[Daniel Pacak]

Since we have limited capacity to support managed registries and we already have #279 to track integration with GCR the only thing I can suggest for now is to follow the contributing guide and debug the operator code out of cluster to find the root cause of the error.

[dgdevops]

Thank you for your response @danielpacak, I will look into it.

[Daniel Pacak]

In particular, check parser logic from ImagePullSecrets in the MapDockerRegistryServersToAuths method.

[Daniel Pacak]

Closing as a duplicate of #279