Creating AWS4Auth instances with STS tokens leaks memory, slows down over time

[Miikka Koskinen]

If you continuously create AWS4Auth instances in with the security_token argument set, it will slowly leak memory and make request signing slower.

Our production service creates a new AWS4Auth instance for every request to AWS S3 (possibly we should just re-use them) and we noticed that after tens of thousands of requests, the requests were getting slower and slower. Restarting the service makes them fast again. Looks like the code below is causing the issue:

httpx_auth/httpx_auth/aws.py

Lines 50 to 53 in e3bd739

 # TODO Check if we really need to be able to override this default ? 
 if self.security_token: 
     # TODO Avoid modifying shared variable 
     self.default_include_headers.append("x-amz-security-token") 

Every time you create a new AWS4Auth instance, one more copy of x-amz-security-token gets appended to default_include_headers. Here's a Python REPL example demonstrating the problem:

>>> from httpx_auth.aws import AWS4Auth
>>> AWS4Auth("test", "test", "us-east-1", "s3", security_token="token").default_include_headers
['host', 'content-type', 'date', 'x-amz-*', 'x-amz-security-token']
>>> AWS4Auth("test", "test", "us-east-1", "s3", security_token="token").default_include_headers
['host', 'content-type', 'date', 'x-amz-*', 'x-amz-security-token', 'x-amz-security-token']
>>> AWS4Auth("test", "test", "us-east-1", "s3", security_token="token").default_include_headers
['host', 'content-type', 'date', 'x-amz-*', 'x-amz-security-token', 'x-amz-security-token', 'x-amz-security-token']

[Colin Bounouar]

Thanks for reporting this issue. As you pointed out, the auth instances should be reused as much as possible if you do not change any parameter to it.
In any case this is an issue that will be tackled of course !

[Colin Bounouar]

The fix is now on the develop branch and will be part of the next httpx-auth release.