Performance issue due to SSLContext being recreated over and over
mm-matthias opened this issue · comments
Describe the bug
botocore
request/response times are much slower than expected in certain scenarios due to unexpected SSL overhead.
Expected Behavior
botocore /boto3 performs fast for all requests.
Ideally the first request does not take longer than any following requests (e.g. the SSLContext could be initialized during module init or be triggered by the user of the library in some way before any requests happen).
Current Behavior
Request/responses are too slow, see below.
Reproduction Steps
botocore performance is much slower than expected in certain scenarios. This can easily be reproduced by
import boto3
import time
session = boto3.session.Session()
for i in range(10):
start = time.time()
session.resource("s3").Object("mybucket", "mykey").get()["Body"].read()
print(time.time() - start)
which yields
0.4858591556549072
0.1862480640411377
0.20270681381225586
0.22403812408447266
0.19844722747802734
0.18699288368225098
0.17718195915222168
0.18413686752319336
0.1918637752532959
0.18160700798034668
This can be optimized by sharing the resource between iterations:
import boto3
import time
session = boto3.session.Session()
resource = session.resource("s3")
for i in range(10):
start = time.time()
resource.Object("mybucket", "mykey").get()["Body"].read()
print(time.time() - start)
which yields
0.25886106491088867
0.05055093765258789
0.050694942474365234
0.05214500427246094
0.05020618438720703
0.05117917060852051
0.04602789878845215
0.048911094665527344
0.048056840896606445
0.04799509048461914
One can see that the initial request is always quite slow.
But the issue goes further. If there is some delay between requests all the requests get slow. This can be reproduced with:
import boto3
import time
session = boto3.session.Session()
resource = session.resource("s3")
for i in range(10):
start = time.time()
resource.Object("mybucket", "mykey").get()["Body"].read()
print(time.time() - start)
time.sleep(10)
This yields
0.22714900970458984
0.17821383476257324
0.21504521369934082
0.17998909950256348
0.2338550090789795
0.19853711128234863
0.23908185958862305
Switching to sleep(5)
yields fast requests again (except for the first one).
The reason for the slowness is that
- the http keep-alive timeout that botocore uses implicitly via urllib3 seems to be rather low (probably between 5 and 10 seconds) and there doesn't seem to be a straightforward way to increase it. This causes an increased number of connection creations and ssl handshakes the as well as the issue mentioned in the next point.
- enabling debug logging for urllib3 suggests that
is_connection_dropped
returnsTrue
for the connections that are issue after asleep(10)
. So presumably something (the s3 server?) closed the connection. In this case there might not be a lot of things that botocore could do, except for the next point about making SSL connection creation faster.
- enabling debug logging for urllib3 suggests that
- initializing SSL connections has become a lot slower in recent python versions, see python/cpython#95031 . Other major libraries like
requests
are also affected, see psf/requests#6667 .
The slow requests are slow, becauseload_verify_locations
is called once for each of them. This happens through the urllib3PoolManager
/HTTPConnectionPool
and the recreation ofSSLContext
.
You can verify this by running any of the above reproducers in a profiler and look at the number of calls to/time spent inload_verify_locations
. - The problem is even exacerbated when using a proxy as this leads to even more SSL calls. Adding different hosts to this makes the problem even worse as this leads to more HTTPConnectionPools being created.
Possible Solution
- The solution for the http keep-alive timeout is to make this configurable somehow so that more requests can benefit from connection pooling.
- The solution to avoid the overhead from
load_verify_locations
is probably to share theSSLContext
across multiple invocations/threads. Therequests
issue I've mentioned above has a proposed workaround.
Additional Information/Context
No response
SDK version used
1.34.98
Environment details (OS name and version, etc.)
MacOS & Linux
Python 3.12.3
Thanks for reporting this issue — after discussing with the team we decided that we should continue tracking this for further review and investigation.