Latest version crashes on ECS fargate

[Ho-Yon Mak]

Environment

• Elixir & Erlang versions (elixir --version): 1.13.2
• ExAws version mix deps |grep ex_aws: 2.3.0
• HTTP client version. IE for hackney do mix deps | grep hackney: 1.18.1

Current behavior

Running an elixir app with ex_aws on ECS fargate crashes on startup when using the latest version of ex_aws.

Error message:

GenServer ExAws.InstanceMetaTokenProvider terminating
** (RuntimeError) Instance Meta Error: {:error, %{reason: :checkout_failure}}

You tried to access the AWS EC2 Instance Metadata token API, but it could not be reached.

Please check AWS EC2 Instance Metadata Service configuration to make sure the service is enabled.

    (ex_aws 2.3.0) lib/ex_aws/instance_meta_token_provider.ex:94: ExAws.InstanceMetaTokenProvider.request_token/1
    (ex_aws 2.3.0) lib/ex_aws/instance_meta_token_provider.ex:57: ExAws.InstanceMetaTokenProvider.refresh_token/2
    (ex_aws 2.3.0) lib/ex_aws/instance_meta_token_provider.ex:47: ExAws.InstanceMetaTokenProvider.handle_call/3
    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4
    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6
    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message (from ExAws.Config.AuthCache): {:refresh_token, %{access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], host: "s3.eu-west-2.amazonaws.com", http_client: ExAws.Request.Hackney, json_codec: Jason, normalize_path: true, port: 443, region: "eu-west-2", retries: [max_attempts: 10, base_backoff_in_ms: 10, max_backoff_in_ms: 10000], scheme: "https://", secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]}}

I think this is because of the new IMDSv2 support added in this pr which crashes as the IMDSv2 endpoint isn't available on ECS fargate.

This command from the docs doesn't work on ECS fargate for example:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

Instead AWS_CONTAINER_CREDENTIALS_RELATIVE_URI should be used as in ExAws.InstanceMeta

Expected behavior

The app starts successfully without error and credentials are fetched using the existing AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable and doesn't attempt to use IMDSv2.

[Robin Fourcade]

If you guys have the same issue with ECS on EC2, here's a way to fix this (with IMDSv2 best practices):

resource "aws_instance"  ... {
  metadata_options {
    http_tokens                 = "required"
    http_endpoint               = "enabled"
    http_put_response_hop_limit = 2 # https://stackoverflow.com/a/71884476/5692012
  }
}