Giters

awslabs / amazon-ecr-credential-helper

Automatically gets credentials for Amazon ECR on docker push/docker pull

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

gitlabrunner + docker missing region

lorenzstorm1 opened this issue · comments

commented

Hi,

following Setup:

  • Gitlab Runner and Docker installed on EC2 Amazon Linux
  • EC2 has Role with permissions to read from private ECR attached
  • amazon-ecr-credential-helper installed and configured

Everything configured like in the Documentation but still not working as it should. I keep getting this error from the Pipeline:
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://AccountID.dkr.ecr.eu-central-1.amazonaws.com/v2/aws-tools/manifests/latest": no basic auth credentials (manager.go:237:0s)

Details on Configuration:

cat /etc/gitlab-runner/config.toml
concurrent = 2
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ip-XX-XX-XXX-XX.eu-central-1.compute.internal"
  limit = 3
  url = "https://gitlab.xxxx.net/"
  id = 560
  token = "xxxxxxxxxxxxx"
  token_obtained_at = 2023-05-24T07:03:10Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\"  }"]
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "debian:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    network_mode = "host"
    shm_size = 0

sh-4.2$ docker-credential-ecr-login version 
0.6.3

sudo cat /root/.docker/config.json
{
 "credsStore": "ecr-login" 
}

cat .gitlab-ci.yml
unit-test:
  stage: unit-test
  image: $CONTAINER

cat variables.yml
  CONTAINER: "AccountID.dkr.ecr.eu-central-1.amazonaws.com/aws-tools:latest"

interesting part of below log: failed to resolve service endpoint, an AWS region is required, but was not found

cat /root/.ecr/ecr.log
time="2023-05-24T07:37:48Z" level=debug msg="Listing credentials" 
time="2023-05-24T07:37:48Z" level=debug msg="Checking file cache" registry= 
time="2023-05-24T07:37:48Z" level=debug msg="Calling ECR.GetAuthorizationToken for default registry" 
time="2023-05-24T07:37:48Z" level=debug msg="couldn't get authorization token for default registry" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to resolve service endpoint, an AWS region is required, but was not found"
time="2023-05-24T07:37:48Z" level=debug msg="Checking file cache for ECR Public" 
time="2023-05-24T07:37:48Z" level=debug msg="couldn't get authorization token for public registry" error="ecr: failed to get authorization token: operation error ECR PUBLIC: GetAuthorizationToken, https response error StatusCode: 400, RequestID: e90e44fd-c8dc-407f-97ea-2334d28f9ab6, api error AccessDeniedException: User: arn:aws:sts::<AccountID>:assumed-role/gitlab-ami-host-ec2-role/i-0b8faa1c90994929f is not authorized to perform: ecr-public:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr-public:GetAuthorizationToken action" 
time="2023-05-24T07:37:48Z" level=debug msg="Listing credentials" 
time="2023-05-24T07:37:48Z" level=debug msg="Checking file cache" registry= 
time="2023-05-24T07:37:48Z" level=debug msg="Calling ECR.GetAuthorizationToken for default registry" 
time="2023-05-24T07:37:48Z" level=debug msg="couldn't get authorization token for default registry" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to resolve service endpoint, an AWS region is required, but was not found" 
time="2023-05-24T07:37:48Z" level=debug msg="Checking file cache for ECR Public" 
time="2023-05-24T07:37:49Z" level=debug msg="couldn't get authorization token for public registry" error="ecr: failed to get authorization token: operation error ECR PUBLIC: GetAuthorizationToken, https response error StatusCode: 400, RequestID: 636d4a9c-186d-4c56-aab5-9e13f1897cdd, api error AccessDeniedException: User: arn:aws:sts::<AccountID>:assumed-role/gitlab-ami-host-ec2-role/i-0b8faa1c90994929f is not authorized to perform: ecr-public:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr-public:GetAuthorizationToken action"
commented

What is interesting, after running a docker pull from the console of the gitlab runner (which works), the auth is cached in /root/.ecr/cache.json and gitlab runner then also can authenticate using that cached file.

Docker pull from console:

time="2023-05-24T07:38:35Z" level=debug msg="Retrieving credentials" region=eu-central-1 registry=<AccountID>  serverURL=<AccountID> .dkr.ecr.eu-central-1.amazonaws.com service=ecr 
time="2023-05-24T07:38:35Z" level=debug msg="Checking file cache" registry=<AccountID>  
time="2023-05-24T07:38:35Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=<AccountID> time="2023-05-24T07:38:35Z" level=debug msg="Saving credentials to file cache" registry=<AccountID>  service=ecr

Docker pull from gitlab runner:
interesting part of below log:
failed to resolve service endpoint, an AWS region is required, but was not found
Using cached token" registry=AccountID

time="2023-05-24T07:36:36Z" level=debug msg="Listing credentials" 
time="2023-05-24T07:36:36Z" level=debug msg="Checking file cache" registry= 
time="2023-05-24T07:36:36Z" level=debug msg="Calling ECR.GetAuthorizationToken for default registry" 
time="2023-05-24T07:36:36Z" level=debug msg="couldn't get authorization token for default registry" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to resolve service endpoint, an AWS region is required, but was not found" 
time="2023-05-24T07:36:36Z" level=debug msg="Checking file cache for ECR Public" 
time="2023-05-24T07:36:37Z" level=debug msg="couldn't get authorization token for public registry" error="ecr: failed to get authorization token: operation error ECR PUBLIC: GetAuthorizationToken, https response error StatusCode: 400, RequestID: 8d80d762-f674-4f44-b0c9-9ae8ded11862, api error AccessDeniedException: User: arn:aws:sts::<AccountID>:assumed-role/gitlab-ami-host-ec2-role/i-0b8faa1c90994929f is not authorized to perform: ecr-public:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr-public:GetAuthorizationToken action" 
time="2023-05-24T07:36:37Z" level=debug msg="Retrieving credentials" region=eu-central-1 registry=<AccountID> serverURL="https://<AccountID>.dkr.ecr.eu-central-1.amazonaws.com" service=ecr 
time="2023-05-24T07:36:37Z" level=debug msg="Checking file cache" registry=<AccountID> 
time="2023-05-24T07:36:37Z" level=debug msg="Using cached token" registry=<AccountID> 
time="2023-05-24T07:36:37Z" level=debug msg="Listing credentials" 
time="2023-05-24T07:36:37Z" level=debug msg="Checking file cache" registry= 
time="2023-05-24T07:36:37Z" level=debug msg="Calling ECR.GetAuthorizationToken for default registry" 
time="2023-05-24T07:36:37Z" level=debug msg="couldn't get authorization token for default registry" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to resolve service endpoint, an AWS region is required, but was not found" 
time="2023-05-24T07:36:37Z" level=debug msg="Checking file cache for ECR Public" 
time="2023-05-24T07:36:37Z" level=debug msg="couldn't get authorization token for public registry" error="ecr: failed to get authorization token: operation error ECR PUBLIC: GetAuthorizationToken, https response error StatusCode: 400, RequestID: ab2361a4-88ac-4001-8e5e-ae88e097af41, api error AccessDeniedException: User: arn:aws:sts::<AccountID>:assumed-role/gitlab-ami-host-ec2-role/i-0b8faa1c90994929f is not authorized to perform: ecr-public:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr-public:GetAuthorizationToken action" 
time="2023-05-24T07:36:37Z" level=debug msg="Retrieving credentials" region=eu-central-1 registry=<AccountID> serverURL="https://<AccountID>.dkr.ecr.eu-central-1.amazonaws.com" service=ecr 
time="2023-05-24T07:36:37Z" level=debug msg="Checking file cache" registry=<AccountID> 
time="2023-05-24T07:36:37Z" level=debug msg="Using cached token" registry=<AccountID>
commented

Found a solution to my Problem:
sudo mkdir /root/.docker
sudo vi /root/.docker/config.json
{ 
 "credHelpers": {
   "ACCOUNTID.dkr.ecr.REGION.amazonaws.com": "ecr-login"
 }
}