aioboto3 9.3.1 regression on long-running tasks
dacevedo12 opened this issue · comments
- Async AWS SDK for Python version:
aioboto3: 9.3.1
,aiobotocore: 1.0.4
,boto3: 1.20.24
,botocore: 1.23.24
,aiohttp: 3.8.1
- Python version: 3.9
- Operating System: nixos
Description
After updating from aioboto3 8.0.3 I got some errors with long-running tasks that execute many requests to dynamodb, such as:
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: Response payload is not completed
,
ConnectionResetError: Cannot write to closing transport
and
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: [Errno None] Can not write request body for https://dynamodb.us-east-1.amazonaws.com/
What I Did
I'm not sure exactly what triggered the errors other than using aioboto3 to perform multiple dynamodb operations
my current setup looks like this
RESOURCE_OPTIONS = {
"service_name": "dynamodb",
"region_name": "us-east-1",
}
SESSION = aioboto3.Session()
async def query(...):
async with SESSION.resource(**RESOURCE_OPTIONS) as resource:
table_resource = await resource.Table("my_table")
response = await table_resource.query(...)
items = response.get("Items", [])
while response.get("LastEvaluatedKey"):
response = await table_resource.query(...)
items += response.get("Items", [])
return items
Note:
- I also reuse that
SESSION
constant on other functions for operations likeput_item
, etc. - Downgrading to aioboto3 8.0.3 resolved the problem
@terrycain looks like a regression on aiobotocore?
It seems aiobotocore 2.2.0 reintroduced a retry mechanism that could help solve this
Related #259
v9.5.0 should be out, try that :)