Please consider using GetParameter(s) instead of GetParametersByPath
nachmore opened this issue · comments
We have encountered cases where deploying chamber at scale runs into AWS API limits.
GetParametersByPath has a maximum tps of 100 with customers able to increase this slightly by opening a support case with AWS. GetParameter and GetParameters has a maximum tps of 3,000 and is more suited for high throughput use cases (ref).
In general some kind of caching layer reducing the calls to GetParametersByPath might be useful to allow chamber to be deployed at scale - and will likely improve overall performance.
Chamber uses GetParametersByPath to find all secrets (parameters) for a given service. It's used primarily for loading secrets into environment variables for ensuing commands, but it's also used for export. Based on git history, DescribeParameters used to be employed but was swapped out for GetParametersByPath to take advantage of higher API limits (ironically). The current limit for GetParametersByPath is indeed 100 tps, but I'm not sure what the limit is for DescribeParameters. Based on this SSM doc it might be 40.
I looked at GetParameters, and it requires a list of ARNs for the parameters to get. Unless it allows wildcards - I tried and can't get them to work - then it won't work as a substitute, because chamber doesn't already know the ARNs of the secrets it's looking for.
Caching makes sense, but it may be better to have that layer outside chamber. Another tactic is to set retries and (coming up soon) using the adaptive retry mode for AWS calls; either of those may cope with rate limits to an extent.
Thanks for the follow up! The limits for DescribeParameters are very low - I'll follow up to see if those can be documented publicly. Is there an existing pattern of caching external data (like this) outside of Chamber?