c7n-org MFA issues

Question

c7n-org MFA issues

kwcrook opened this issue 6 years ago · comments

Kevin Crook commented 6 years ago

When run against one single account at a time, c7n-org does great:

accounts.yml:

accounts:

account_id: 'XXXXXXXXXX'
name: XXXXXXXXXXXX
regions:
- us-west-2
  role: arn:aws:iam::XXXXXXXXXX:role/AccountAdministrator
  tags:
- type:dev

c7n-org has trouble dealing with multiple accounts that require MFA:

ClientError: An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,200: custodian.output:ERROR Error while executing policy Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/c7n/policy.py", line 305, in run resources = self.policy.resource_manager.resources() File "/usr/local/lib/python2.7/site-packages/c7n/query.py", line 397, in resources resources = self.augment(self.source.resources(query)) File "/usr/local/lib/python2.7/site-packages/c7n/query.py", line 210, in resources return self.query.filter(self.manager, **query) File "/usr/local/lib/python2.7/site-packages/c7n/query.py", line 74, in filter client = local_session(self.session_factory).client( File "/usr/local/lib/python2.7/site-packages/c7n/utils.py", line 250, in local_session s = factory() File "/usr/local/lib/python2.7/site-packages/c7n/credentials.py", line 40, in __call__ region or self.region, self.external_id) File "/usr/local/lib/python2.7/site-packages/c7n/credentials.py", line 87, in assumed_session metadata=refresh(), File "/usr/local/lib/python2.7/site-packages/c7n/credentials.py", line 78, in refresh session.client('sts').assume_role, **parameters)['Credentials'] File "/usr/local/lib/python2.7/site-packages/c7n/utils.py", line 347, in _retry return func(*args, **kw) File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call raise error_class(parsed_response, operation_name) ClientError: An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,202: c7n_org:ERROR Exception running policy:elb-tag-compliance account:XXXXXXXXXXXXX region:us-east-1 error:An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,204: c7n_org:ERROR Exception running policy:elb-tag-compliance account:XXXXXXXXXXXXXXX region:us-west-2 error:An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,205: c7n_org:ERROR Exception running policy:elb-tag-compliance account:XXXXXXXXXXXXXXXX region:us-west-2 error:An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,248: c7n_org:INFO Policy resource counts Counter()

When run against all 17 accounts, it eats up all the available memory on my server, causes a kernel panic, and kicks me out of my ssh session.

Kapil Thangavelu · Answer 1 · Sat Jan 20 2018 22:41:42 GMT+0800 (China Standard Time)

wow, thats interesting, thanks for the bug report. there's a --debug flag on org to go single threaded through multiple accounts (it also drops into a debugger on error). it sounds like your just running this from an ec2 instance so instance role/profile as initial credential source? does it go haywire from the get go, or after some time?

Kevin Crook · Answer 2 · Tue Jan 23 2018 00:28:50 GMT+0800 (China Standard Time)

It goes haywire from the get go. It is running on an ec2 instance, but per the boto3 documentation, it should look at the credential file in .aws first, and that's what it does when only run one at a time. I'm going to try the --debug flag and see what it kicks out.

Kapil Thangavelu · Answer 3 · Tue Jan 23 2018 02:38:02 GMT+0800 (China Standard Time)

are you putting static credentials in ~/.aws or are you doing a login with mfa.. perhaps it worth coming up a level, how do you do automate requiring sts w/ mfa automation on a cron execution?. typical is to just setup instance profile/arn for credential source sans the mfa.

Kevin Crook · Answer 4 · Tue Jan 23 2018 06:47:58 GMT+0800 (China Standard Time)

We're not yet able to automate sts w/ mfa. So far cloud custodian runs with temporary credentials from whoever is currently developing policies.

We use a python wrapper around the aws-adfs package because it supports MFA from the CLI. This talks to an ADFS server which authenticates our credentials against a CAS server. From the exchange between ADFS and AWS, unique access keys (per account) are granted along with a single session token (the adfs_auth token). Our python writes these unique keys to the .aws credentials and config folder as shown below:

.aws/config
`[dev-account-1]
region = us-west-2
adfs_role = Administrator@dev-account-1
adfs_expires = MM-DD-YYYY HH:MM
adfs_netid = MyOrgUsername

[prd-account-1]
region = us-west-2
adfs_role = Administrator@prd-account-1
adfs_expires = MM-DD-YYYY HH:MM
adfs_netid = MyOrgUsername

[prd-account-2]
region = us-west-2
adfs_role = Administrator@prd-account-2
adfs_expires = MM-DD-YYYY HH:MM
adfs_netid = MyOrgUsername`

.aws/credentials
`[dev-account-1]
aws_access_key_id = ASIAUNIQUETEMPCREDENTIAL
aws_secret_access_key = MzVtCTSv27Rdl9ffda
aws_session_token = FQoDYXdzEB8aDGQ/PMr54ItZCKRAumnT2k+x2vkDZJzrlkeKpe0lzro9phW/N9QO/ncz9Rcz+iiMzo1VSDYznlh83vH0+KJ3eatbOaUhxb7804UECmMRF1==

[prd-account-1]
aws_access_key_id = ASIAUNIQUETEMPCREDENTIAL
aws_secret_access_key = P0JSeENGFYaj8qMzV
aws_session_token = FQoDYXdzEDIaDGNB4ZLzfSKRAl6R0qZpEEDh4As+UvM3JaLqaY7kMZ4Np/kVWX1x2rkB2Q9vYQiG79AqAtkB0RV1UMb9jyd1JZQgmWL4IJaA7j/mFR0==

[prd-account-2]
aws_access_key_id = ASIAUNIQUETEMPCREDENTIAL
aws_secret_access_key = SNdN5FgCbXx/4R4CMbi7J5A
aws_session_token = FQoDYXdzEDIaDPOhg37K7+SKRAppDgTIZ6Bz9IoavVNuOjyComZecNYQVSEdFPh2x3nkBv7YEVurwcPLNVI75vKUguJT9ssDzU++d5dFQgAPV==

[ . . . ]

[all]
adfs_auth = UBER_long_auth_token==

`

Kevin Crook · Answer 5 · Thu Jan 25 2018 07:14:33 GMT+0800 (China Standard Time)

It seems that c7n-org only gets credentials out of the [default] section of the credentials and config files. Does this mean that c7n-org will only work if you have cross-account roles set up?

Kapil Thangavelu · Answer 6 · Wed Jan 31 2018 07:12:39 GMT+0800 (China Standard Time)

yes, its not setup to lookup a profile for each account, the creds found for the cli execution (sdk defaults, --profile, --assume cli options) are expected to be able to sts role assume to the other roles. it seems reasonable as an enhancement to allow for the profile option to be specified on each account.

Kevin Crook · Answer 7 · Thu Feb 01 2018 00:58:15 GMT+0800 (China Standard Time)

My current workaround: renaming the profile for each account to default one at a time. While it is tedious, it is much faster than needing to accept push notifications to my phone every minute.

My end goal would be to have something like this automated so it can run nightly on a cron job. I'm testing a possible workaround by setting up an instance profile that has access to assume cross-account roles, but this still hasn't worked.

Kapil Thangavelu · Answer 8 · Wed Feb 14 2018 08:31:18 GMT+0800 (China Standard Time)

for tools/c7n_guardian (uses c7n-org under the hood) i ended up adding in support for profile as part of the account config, should be easy to up-port it to c7n-org

Kapil Thangavelu · Answer 9 · Sun Mar 18 2018 00:08:50 GMT+0800 (China Standard Time)

ported over to c7n-org that should allow for the mfa case if you active credentials against those profiles. basically in c7n-org config instead of role specify profile key for each account.

sandeepreddy chintapally · Answer 10 · Tue Nov 06 2018 04:00:48 GMT+0800 (China Standard Time)

Hi kaplit i need help on the same issue, i want to use profiles name in the c7n-org command will it possible