Empty response when running `list_objects_v2`

Question

Empty response when running `list_objects_v2`

aayushgupta899 opened this issue a year ago · comments

Describe the bug
I am running the list_objects_v2 using aiobotocore, in a docker container on Amazon EMR. The following code snippet returns an empty response:

import asyncio
import aiobotocore
from aiobotocore.session import get_session

BUCKET_NAME = "test-bucket-20230713"

async def list_objects():
    session = get_session()

    async with session.create_client('s3', region_name='us-east-1') as client:
        response = await client.list_objects_v2(Bucket=BUCKET_NAME)
        print(response)

        for obj in response.get('Contents', []):
            print(obj['Key'])

loop = asyncio.run(list_objects())

Output:

{'ResponseMetadata': {'RequestId': 'QJ9FWRY2HJT32QAT', 'HostId': 'iplqiXNK3ot49VwoMyt6xnzH16aQXIUHLlQGA8PNFiiaXY2DcGcucz3cV9GUq3co6Mw6MnCmmA4=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'iplqiXNK3ot49VwoMyt6xnzH16aQXIUHLlQGA8PNFiiaXY2DcGcucz3cV9GUq3co6Mw6MnCmmA4=', 'x-amz-request-id': 'QJ9FWRY2HJT32QAT', 'date': 'Mon, 17 Jul 2023 20:13:04 GMT', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}}

Stack trace: stacktrace_aiobotocore.txt

Same command using botocore:

import botocore.session

BUCKET_NAME = "test-bucket-20230713"

session = botocore.session.get_session()
client = session.create_client('s3')
client.list_objects_v2(Bucket=BUCKET_NAME)

Output:

{'ResponseMetadata': {'RequestId': '4HZWK0QADQVM3T89', 'HostId': 'ogbCYrdCnE0Wse2dEolR/1MuF1lDqAprZ0yn1WyqgNiByFSqhNo8A2nvrHaLI+EgtkG9AzXFIr0=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'ogbCYrdCnE0Wse2dEolR/1MuF1lDqAprZ0yn1WyqgNiByFSqhNo8A2nvrHaLI+EgtkG9AzXFIr0=', 'x-amz-request-id': '4HZWK0QADQVM3T89', 'date': 'Mon, 17 Jul 2023 21:10:08 GMT', 'x-amz-bucket-region': 'us-east-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Contents': [{'Key': 'data/test2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 10, tzinfo=tzlocal()), 'ETag': '"2f99a26fd3660ba94188d1b7106eda6a"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 58, tzinfo=tzlocal()), 'ETag': '"e98a3e683a5fb991a3aecb5f42bb70cd"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_old.txt', 'LastModified': datetime.datetime(2023, 7, 13, 18, 53, 26, tzinfo=tzlocal()), 'ETag': '"7a1ff55a2596e4d015683d012c9cc7ca"', 'Size': 0, 'StorageClass': 'STANDARD'}], 'Name': 'test-bucket-20230713', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 3}

Stack trace: stacktrace_botocore.txt

I tried this with an older version of aiobotocore(==2.1.0), and I get the same issue. However, when I try with older versions of botocore (==1.27.96) and boto3 (1.24.96), it works:

import asyncio
import aiobotocore
from aiobotocore.session import get_session

BUCKET_NAME = "test-bucket-20230713"

async def list_objects():
    session = get_session()

    async with session.create_client('s3', region_name='us-east-1') as client:
        response = await client.list_objects_v2(Bucket=BUCKET_NAME)
        print(response)

        for obj in response.get('Contents', []):
             #print(obj['Key'])

loop = asyncio.run(list_objects())

Output:

{'ResponseMetadata': {'RequestId': 'PRE2VG4B2A3S2392', 'HostId': 'XvXd9+k5PlfKuQpIGVEaWPgK4pNM1kAM9JixYJ1tWLeOKwUvO1tF2ABnPRDu3Gx80igP0Itjv7w=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'XvXd9+k5PlfKuQpIGVEaWPgK4pNM1kAM9JixYJ1tWLeOKwUvO1tF2ABnPRDu3Gx80igP0Itjv7w=', 'x-amz-request-id': 'PRE2VG4B2A3S2392', 'date': 'Mon, 17 Jul 2023 21:41:03 GMT', 'x-amz-bucket-region': 'us-east-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Contents': [{'Key': 'data/test2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 10, tzinfo=tzlocal()), 'ETag': '"2f99a26fd3660ba94188d1b7106eda6a"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 58, tzinfo=tzlocal()), 'ETag': '"e98a3e683a5fb991a3aecb5f42bb70cd"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_old.txt', 'LastModified': datetime.datetime(2023, 7, 13, 18, 53, 26, tzinfo=tzlocal()), 'ETag': '"7a1ff55a2596e4d015683d012c9cc7ca"', 'Size': 0, 'StorageClass': 'STANDARD'}], 'Name': 'test-bucket-20230713', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 3}
data/test2.bin
data/test_2.bin
data/test_old.txt

Checklist

I have reproduced in environment where pip check passes without errors
I have provided pip freeze results
I have provided sample code or detailed way to reproduce
I have tried the same code in botocore to ensure this is an aiobotocore specific issue
I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.4.2
aiohttp==3.7.4
aioitertools==0.5.1
amzn-pyodin==1.2
async-timeout==3.0.1
attrs==23.1.0
boto3==1.28.2
botocore==1.31.2
certifi==2021.5.30
chardet==4.0.0
charset-normalizer==2.0.7
docutils==0.15.2
fsspec==2023.4.0
future==0.18.1
gitdb==0.6.4
GitPython==2.1.8
hnswlib==0.3.4
idna==3.2
idna-ssl==1.0.0
importlib-metadata==6.0.0
jmespath==0.9.5
joblib==1.2.0
multidict==5.1.0
numpy==1.18.4
pandas==1.1.3
patsy==0.5.1
pip==21.0.1
pybind11==2.3.0
PyGreSQL==5.0.7
python-dateutil==2.8.2
pytz==2023.3
PyYAML==5.4.1
requests==2.28.1
s3fs==2023.6.0
s3transfer==0.6.1
scikit-learn==0.22
scipy==1.3.1
setuptools==57.4.0
six==1.16.0
sklearn==0.0
smmap==0.9.0
statsmodels==0.11.1
toml==0.10.2
torch==1.7.1
tqdm==4.64.0
typing-extensions==4.1.1
urllib3==1.26.13
wrapt==1.11.0
yarl==1.7.2
zipp==3.15.0

Environment:

Python Version: 3.8
OS name and version: Amazon Linux 2

Additional context
This issue was initially encountered while using s3fs.S3FileSystem().ls() (issue) and was isolated to be an aiobotocore issue.

Alexander Mohr · Answer 1 · Tue Jul 18 2023 08:11:04 GMT+0800 (China Standard Time)

I tried it on one of my buckets and its working fine, is there a way to expose it to my private aws account: 848120881190 ?

aayushgupta899 · Answer 2 · Thu Jul 20 2023 03:29:55 GMT+0800 (China Standard Time)

I don't think I will be able to expose it to your account to it as the code is running on a company AWS account. Anything else I can try to help you reproduce the issue?

Alexander Mohr · Answer 3 · Thu Jul 20 2023 04:48:58 GMT+0800 (China Standard Time)

ya if you could describe how to setup my account to repro this, IAM, bucket config, etc.

Alexander Mohr · Answer 4 · Mon Aug 07 2023 11:54:28 GMT+0800 (China Standard Time)

btw could you fix your log messages? it's spewing a bunch of errors because the socket it's trying to send is already closed

Alexander Mohr · Answer 5 · Wed Aug 30 2023 00:41:26 GMT+0800 (China Standard Time)

another thing would help if you could repro with moto

aayushgupta899 · Answer 6 · Tue Oct 24 2023 08:17:50 GMT+0800 (China Standard Time)

This started working after I upgraded to aiobotocore==2.5.0. Not sure what changed though. Here are my pip freeze results:

aiobotocore==2.5.0
aiohttp==3.8.5
aioitertools==0.5.1
aiosignal==1.2.0
amzn-pyodin==1.2
async-timeout==4.0.2
attrs==23.1.0
boto3==1.28.65
botocore==1.31.65
charset-normalizer==3.0.0
docutils==0.15.2
frozenlist==1.3.3
fsspec==2022.2.0
gitdb==0.6.4
GitPython==2.1.8
hnswlib==0.3.4
idna==3.2
importlib-metadata==4.6.1
jmespath==0.9.5
joblib==1.2.0
multidict==6.0.4
numpy==1.18.4
pandas==1.1.3
patsy==0.5.1
pip==21.0.1
pybind11==2.3.0
PyGreSQL==5.0.7
python-dateutil==2.8.2
pytz==2023.3
s3fs==2022.2.0
s3transfer==0.7.0
scikit-learn==0.22
scipy==1.3.1
setuptools==57.4.0
six==1.16.0
sklearn==0.0
smmap==0.9.0
statsmodels==0.11.1
toml==0.10.2
tqdm==4.64.0
typing-extensions==4.1.1
urllib3==1.26.13
wrapt==1.15.0
yarl==1.8.2
zipp==3.15.0

aayushgupta899 · Answer 7 · Tue Oct 24 2023 08:20:14 GMT+0800 (China Standard Time)

Closing this as this is no longer an issue. Please let me know if there are any more details which are required to be added for this.