credentials precedence in release >= 3.2.1297.0

[jorditpuig]

Hi,

I suspect that release 3.2.1297.0 introduced a, perhaps involuntary, change in the credentials precedence logic. What we are experiencing in all our EC2 managed instances is that the agent (for instance when running AWS-RunPatchBaseline) is using the default profile credentials from /root/.aws/credentials instead of the EC2 instance profile role. Versions before 3.2.1297.0 were using the instance profile role although there was a /root/.aws/credentials present.

To make things more confusing, the documentation states that the precedence should indeed be /root/.aws/credentials first and EC2 instance profile role last but only for "SSM Agent version 3.1.1927.0 or earlier, and for Amazon ECS container instances". In our case none of the instances is an ECS container instance and all of them are running amazon-ssm-agent version 3.2.1297.0.

Furthermore, in the same documentation page it is clearly stated that (highlighting is mine)

Starting with version 3.2.183.0 of SSM Agent, the agent stores a set of temporary credentials at /var/lib/amazon/ssm/credentials (for Linux and macOS) or %PROGRAMFILES%\Amazon\SSM\credentials (for Windows Server). The temporary credentials have the permissions you specify for the IAM role you chose for Default Host Management Configuration, or the instance profile attached to your managed node.

This part of the documentation is not mentioning /root/.aws/credentials at all, so I guess starting with version 3.2.183.0 the Default Host Management Configuration or the instance profile role is used.

I understand that the documentation might be out of sync. Nevertheless, releases before 3.2.1297.0 were not using (or not giving higher precedence)/root/.aws/credentials but release 3.2.1297.0 does. So my question is first what is the expected behaviour of release 3.2.1297.0 regarding the credentials precedence and second do you believe that the documentation is reflecting this precedence?

[Gianni Leslie]

Hi, the documentation is pending update but the credential precedence in the latest version is intentionally the same as it was for 3.1 versions. Working with the patch team on how we can ensure that the credential precedence always matches the agent's for future proofing. At the moment, the patch script is following the default credential precedence detailed in our documentation

Note also all 3.2 versions before 3.2.1297.0 have been marked deprecated.

[1257472170]

Hi All
the different credential files' location are challenge to me: we offer the ec2 instance to users and they will configure proguser as the default profile. But the patch job will find the credential in ~/.aws/credentials instead of /var/lib/amazon/ssm/credentials.
and I open the support ticket to ask the following issues have also been confirmed by the background team:
Q1. The ssm agent only needs the iam role policy of the instance, even if the credentials are configured in the aws cli.
Yes, SSM Agent only requires an instance IAM role with appropriate SSM permissions if patching needs to be successful.

Q2. If there is a [default] part in the aws cli credentials in the instance, the customer needs to modify it to a different name, and then can the ssm agent operate with the correct permissions?
Yes, by default the settings are in a config file called "default", so you can delete the [default] config file in the credentials file, or delete that file entirely if you don't need it. It's the solution support recommand, but I still have doubts

Q3. Even if the [default] configuration file has full ssm permissions, the ssm agent will still report an error when running "AWS-RunPatchBaseline", right?
Yes, you'll get "doesn't match the credentials" errors when patching.