Fetch errors since upgrade to 5.13.0
GijsbertVanOeveren opened this issue · comments
Recently we upgraded to scoutsuite 5.13.0 now running on python3.9
Since then we experience various fetch errors we have never encountered before.
Command used: scout aws --report-name ${ACCOUNTALIAS} --debug --report-dir /var/www/html/ -f --logfile log/${ACCOUNTALIAS}.log --quiet --no-browser -r eu-west-1 eu-central us-east-1 --max-workers 3 --debug
Log output:
2023-07-24 12:02:33 scout[3328] ERROR rds.py L168: Failed fetching DB parameters for tf-pg-mariadb-10-6-wikidb: An error occurred (Throttling) when calling the DescribeDBParameters operation (reached max retries: 4): Rate exceeded
Traceback (most recent call last):
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/rds.py", line 168, in _get_and_set_db_parameters
parameters = await AWSFacadeUtils.get_all_pages(
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 27, in get_all_pages
results = await AWSFacadeUtils.get_multiple_entities_from_all_pages(
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 57, in get_multiple_entities_from_all_pages
return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities))
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/utils.py", line 27, in run_concurrently
return await run_function_concurrently(function)
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 57, in
return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities))
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 73, in _get_all_pages_from_paginator
for page in paginator:
File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/paginate.py", line 269, in iter
response = self._make_request(current_kwargs)
File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/paginate.py", line 357, in _make_request
return self._method(**current_kwargs)
File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/client.py", line 534, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/client.py", line 976, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (Throttling) when calling the DescribeDBParameters operation (reached max retries: 4): Rate exceeded
2023-07-24 12:02:39 ip-10-229-14-28 scout[3328] ERROR base.py L17: Failed to call fetch_all() for resource parameter_groups: 'Parameters'
Traceback (most recent call last):
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/base/resources/base.py", line 17, in call
await task
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/parametergroups.py", line 14, in fetch_all
name, resource = self._parse_parameter_group(raw_parameter_group)
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/parametergroups.py", line 20, in _parse_parameter_group
raw_parameter_group['parameters'] = raw_parameter_group.pop('Parameters')
KeyError: 'Parameters'
2023-07-24 12:05:08 ip-10-229-14-28 scout[3328] ERROR base.py L17: Failed to call fetch_all() for resource snapshots: 'Attributes'
Traceback (most recent call last):
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/base/resources/base.py", line 17, in call
await task
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/snapshots.py", line 14, in fetch_all
name, resource = self._parse_snapshot(raw_snapshot)
File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/snapshots.py", line 28, in _parse_snapshot
snapshot['attributes'] = raw_snapshot['Attributes']
KeyError: 'Attributes'
/opt/scoutsuite/bin/scout --version
Scout Suite 5.13.0
/opt/scoutsuite/bin/python --version
Python 3.9.17
/opt/scoutsuite/bin/python3 --version
Python 3.9.17