HTTP proxy configuration for ECS anywhere

[pmv]

It would be an improvement if /doc_source/http_proxy_config.md would address ECS anywhere. I'm not sure if it does not because ECS anywhere only recently went GA, or if there is a better location for ECS anywhere documentation.

Short version:

The documentation for setting NO_PROXY in /etc/systemd/system/docker.service.d/http-proxy.conf
has
Environment="NO_PROXY=169.254.169.254"
but cloudwatch on ECS anywhere does not work for me unless I have
Environment="NO_PROXY=169.254.169.254,169.254.170.2"

Can this be updated?

Extended details:

I was working on setting up ECS anywhere on RHEL 7. The installation script does not install/configure docker on RHEL, so I was looking for documentation on proxy settings, and landed at /doc_source/http_proxy_config.md (among other places).

Using settings on that page, I was able to get the agent up and connected to AWS and run a sample task. However, when trying to integrate cloudwatch into the ECS task definition, I was encountering an odd error.

Jun 14 18:45:48 xxxxx dockerd: time="2021-06-14T18:45:48.948385798-05:00" level=error msg="Failed to create log stream" errorCode=CredentialsEndpointError logGroupName=/ecs/hello-world-ecsanywhere logStreamName=ecs/ecsanywhere/8c735d0cca854d90bd7271bf8a852a9b message="failed to load credentials" origError=": "

Since this is my first experience with AWS / docker / cloudwatch / etc., the error meant nothing to me, but I have had enough history with proxies to wonder if that was the cause. Next I turned on docker debug logging in hopes it would give me more details, but unfortunately nothing more about "origError":

Jun 14 18:45:48 xxxxx  dockerd: time="2021-06-14T18:45:48.939616967-05:00" level=debug msg="Trying to get credentials from awslogs-credentials-endpoint"
Jun 14 18:45:48 xxxxx  dockerd: time="2021-06-14T18:45:48.939657400-05:00" level=debug msg="Created awslogs client" region=us-east-2
Jun 14 18:45:48 xxxxx  dockerd: time="2021-06-14T18:45:48.948385798-05:00" level=error msg="Failed to create log stream" errorCode=CredentialsEndpointError logGroupName=/ecs/hello-world-ecsanywhere logStreamName=ecs/ecsanywhere/8c735d0cca854d90bd7271bf8a852a9b message="failed to load credentials" origError=": "

Luckily, the debug statement led me to here: https://github.com/moby/moby/blob/master/daemon/logger/awslogs/cloudwatchlogs.go#L368, where I found the hardcoded credentials endpoint in the same file: https://github.com/moby/moby/blob/master/daemon/logger/awslogs/cloudwatchlogs.go#L65

After adding that to NO_PROXY, cloudwatch started to work.

Definitely a roundabout way to get where I was trying to go, so if the documentation can be fixed that would be great.

(In hindsight, I see the "/etc/sysconfig/docker" configuration has NO_PROXY as I needed it, but the "/etc/systemd/system/docker.service.d/http-proxy.conf" config does not...ouch)

Additionally, if the error or debug messages would have

1. Done a better job of printing the original error (I assume a socket timeout or could not connect, etc?) and
2. Logged the address of the endpoint it was trying to connect to

That would have helped tremendously. If you feel it's worthwhile to provide that feedback to developers please do (or let me know where the best place to file an issue would be, and I can do it.) Thank you.

[Cameron Sparr]

Hi @pmv, it looks like there could be some configuration issues unrelated to the NO_PROXY env var, could you grab full instance logs with the ecs logs collector and send them to ecs-agent-external at amazon dot com? thanks!

[pmv]

@sparrc - sorry, my original post was a bit long, but I was able to get it working. Trying to be more clear with my feedback:

1. Does https://github.com/awsdocs/amazon-ecs-developer-guide/blob/master/doc_source/http_proxy_config.md?plain=1#L6-L7 need another heading for "ECS Anywhere / other linux proxy configurations". The doc only covers Amazon linux, but ECS Anywhere supports a lot more linux flavors: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html#ecs-anywhere-supported-os
2. Is the config here: https://github.com/awsdocs/amazon-ecs-developer-guide/blob/master/doc_source/http_proxy_config.md?plain=1#L34 valid, or should it be Environment="NO_PROXY=169.254.169.254,169.254.170.2"? In my experience 169.254.170.2 had to be included.

[Josh Bean]

Closing this issue or pull request in advance of archiving this repo. For more information about the decision to archive this repo (and others in the 'awsdocs' org), see the announcement on the AWS News Blog.