Endpoint container doesn't resolve InstanceProfileCredentialsProvider for Fargate execution role

[Chris Hapgood]

My goal is to simulate in a local docker-compose (or docker compose) cluster the same credential acquisition that occurs when running my containers in Fargate (as deployed with the compose-cli).

I am able to start the amazon-ecs-local-container-endpoints in my cluster using docker-compose and I am following the example provided. However, because I want to simulate the Fargate experience, I use the /creds/{role name} model. My application container therefore has the following env vars set:

      AWS_REGION: "us-east-1"
      AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds/pipeline-PlineTaskRole-XC885YZXQBDN"
      ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"

In the amazon-ecs-local-container-endpoints container, essentially copied the example:

  ecs-local-endpoints:
    image: amazon/amazon-ecs-local-container-endpoints
    volumes:
      - /var/run:/var/run
      - $HOME/.aws/:/home/.aws/
    environment:
      HOME: "/home"
      AWS_PROFILE: "default"
    networks:
      credentials_network:
        ipv4_address: "169.254.170.2"

When I start the cluster and attempt to retrieve credentials using DefaultCredentialsProvider from the AWS SDK for Java (version 2.15.57) I get an exception:

Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): The requested metadata is not found at http://169.254.170.2/creds/pipeline-PlineTaskRole-XC885YZXQBDN, InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]

The last entry in that list is telling: it appears the SDK is making the right call, but the endpoints container is not answering.

Addendum: On the assumptions that my default profile was somehow deficient, I attempted to add the (inline) trust policy per the documentation:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::264945611335:user/Administrator"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

However, the AWS IAM console consistently fails to accept the policy and returns a vague This policy contains the following error: Has prohibited field Principal For more information about the IAM policy grammar, see AWS IAM Policies

[Chris Hapgood]

Some debugging conducted from a utility container attached to the credentials network:

bastion:~$ curl http://ecs-local-endpoints/creds
{"AccessKeyId":"ASIAT3L75GJDVMRE33JQ","Expiration":"2021-01-12T18:31:55Z","RoleArn":"","SecretAccessKey":"0rGvdoi3/hbZ/na6b+0WiKnEJwOrD8FkcnqnyBqp","Token":"FwoGZXIvYXdzEAsaDE+Y9dokzy6vEpYa/CKBAQmxf1oDYmLdwFf2FSBNH91FhOr2cp9oR8/O+s/PwbXAahUV+rsDLYZ8yQcFRs7zMCoQ+aswpmcdOA61+BczNvNbZKuRMN6oo5ODM2EspFDsb69vsNum/mwTVIx7yOIH+yvuvfctxtIF7Vvb2kXRmJ+ryRdF3WGxY/WUgUCv5BbIwCiLuvf/BTIoEzvwd8pak28jug1UOIJ76QfG1qFH2qOqok/Z0pQDVkhfoDOowhik0g=="}
bastion:~$ curl http://169.254.170.2/creds/pipeline-PlineTaskRole-XC885YZXQBDN
404 page not found
bastion:~$ curl http://169.254.170.2/v3
Internal Server Error: Failed to list running containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
bastion:~$

[Chris Hapgood]

More information

Collected from the docker host:

[cch1@cch:~/Development/pipeline (master)]$ ls -l  /var/run/docker.sock
lrwxr-xr-x  1 root  daemon  65 Jan  3 16:34 /var/run/docker.sock -> /Users/cch1/Library/Containers/com.docker.docker/Data/docker.sock

And collected from the ecs endpoints container:

sh-4.2# ls -l /var/run/docker.sock 
lrwxr-xr-x 1 root root 65 Jan  3 21:34 /var/run/docker.sock -> /Users/cch1/Library/Containers/com.docker.docker/Data/docker.sock
sh-4.2# cat /var/run/docker.sock 
cat: /var/run/docker.sock: No such file or directory
sh-4.2# 

[Chris Hapgood]

I'm going to close this and open a more pointed issue that addresses the inability to fetch Task Role credentials.