RecursionError through botocore/endpoint_provider.py:572
ilg opened this issue · comments
Describe the bug
Unfortunately, I don't have much in the way of detail nor consistent repro steps, but the stack trace seems to point at botocore. This most likely happened when AWS Secrets Manager rotated a credential, leading to my code's call to create a secrets manager client to get the new secret, and that led into RecursionError: maximum recursion depth exceeded in comparison
(though the code works fine most of the time). This is running inside a docker container on an EC2 instance, all managed with Elastic Beanstalk.
Expected Behavior
No infinite recursion
Current Behavior
[…]
File "/code/[my file].py", line [##], in [function name]
client = boto3.client("secretsmanager", "us-west-2")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/boto3/__init__.py", line 92, in client
return _get_default_session().client(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/boto3/session.py", line 299, in client
return self._session.create_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 997, in create_client
client = client_creator.create_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 161, in create_client
client_args = self._get_client_args(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 520, in _get_client_args
return args_creator.get_client_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/args.py", line 156, in get_client_args
ruleset_resolver = self._build_endpoint_resolver(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/args.py", line 660, in _build_endpoint_resolver
return EndpointRulesetResolver(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/regions.py", line 469, in __init__
self._provider = EndpointProvider(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 704, in __init__
self.ruleset = RuleSet(**ruleset_data, partitions=partition_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 658, in __init__
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 658, in <listcomp>
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
return rule_class(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
return rule_class(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
return rule_class(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
return rule_class(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in __init__
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 533, in <listcomp>
self.rules = [RuleCreator.create(**rule) for rule in rules]
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 572, in create
return rule_class(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint_provider.py", line 445, in __init__
super().__init__(**kwargs)
RecursionError: maximum recursion depth exceeded in comparison
Reproduction Steps
boto3.client("secretsmanager", "us-west-2")
seems pretty self-contained, but works fine the vast majority of the time, so I'm not sure what happened here.
Possible Solution
No response
Additional Information/Context
No response
SDK version used
boto3==1.34.100
Environment details (OS name and version, etc.)
Linux [hostname] 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
FWIW, this does seem to occur every time Secrets Manager rotates the credentials, but only within the already-running web application server. As soon as the worker thread/process restarts, the boto3 client creation call works fine. Given this, I should be able to debug or test further by manually triggering credential rotation.
Thanks for reaching out. When you're able to, please provide a code snippet for reproducing the issue, as well as debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('')
to your script.
Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.