Fail to put on S3 compatible instance

Question

Fail to put on S3 compatible instance

pedro-ricardo opened this issue a year ago · comments

Pedro Ricardo C Souza commented a year ago

Describe the bug
Hello there,
We are testing an S3 compatible storage and we can READ but can not WRITE with aiobotocore in there.
When using the put_object it gets stuck in a long timeout and retry loop, until it breaks.

To narrow it down, we tested the file upload and download in a number of libraries and here is the summary so far:

boto3: ✔️
s3fs: ✖️
aiobotocore: ✖️
s3cmd: ✔️
botocore: ✔️

We plan on using Kedro with this S3 storage and it uses the s3fs that was built on top of aiobotocore, that's how I got here 😁

I'm sending the codes I used to to try to upload a file both in botocore and aiobotocore along with the output related with that PUT method in DEBUG mode.

Checklist

I have reproduced in environment where pip check passes without errors
I have provided pip freeze results
I have provided sample code or detailed way to reproduce
I have tried the same code in botocore to ensure this is an aiobotocore specific issue
I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.5.0
aiohttp==3.8.4
aioitertools==0.11.0
aiosignal==1.3.1
async-timeout==4.0.2
attrs==22.2.0
botocore==1.29.76
charset-normalizer==3.1.0
frozenlist==1.3.3
idna==3.4
jmespath==1.0.1
multidict==6.0.4
python-dateutil==2.8.2
six==1.16.0
typing-extensions==4.5.0
urllib3==1.26.15
wrapt==1.15.0
yarl==1.8.2

Environment

Python 3.8.10
Linux Mint 20.3

Pedro Ricardo C Souza · Answer 1 · Sat Mar 25 2023 06:57:15 GMT+0800 (China Standard Time)

Reproduction Steps

With `aiobocotore`

This code get stuck on the put_object giving lots of timeouts until it breaks.

import asyncio
from aiobotocore.session import get_session

async def go():
    session = get_session()
    
    async with session.create_client('s3',
        endpoint_url=URL,
        region_name=REGION,
        aws_secret_access_key=SECRET_ACCESS_KEY,
        aws_access_key_id=ACCESS_KEY_ID) as client:
        
        # Get Object
        response = await client.get_object( Bucket='test01', Key='hello.txt')
        # Read it
        async with response['Body'] as stream:
            print(await stream.read())

        # Upload Object
        resp = await client.put_object(Bucket='test01', Key='new_hello.txt', Body=b"Hello World\n")
        print(resp)
        
loop = asyncio.get_event_loop()
loop.run_until_complete(go())

With `botocore`

This code works as expected

from botocore.session import get_session

session = get_session()

client = session.create_client('s3',
    endpoint_url=URL,
    region_name=REGION,
    aws_secret_access_key=SECRET_ACCESS_KEY,
    aws_access_key_id=ACCESS_KEY_ID)
    
# Get Object
response = client.get_object(
    Bucket='test01',
    Key='hello.txt')
# Read it
with response['Body'] as stream:
    print(stream.read())

# Upload Object
resp = client.put_object(
    Bucket='test01',
    Key='new_hello.txt',
    Body=b"Hello World\n")
print(resp)

Pedro Ricardo C Souza · Answer 2 · Sat Mar 25 2023 06:57:47 GMT+0800 (China Standard Time)

Expected Behavior

This is the detailed log of the script with bocotore that work as expected:

Calculating signature using v4 auth.
CanonicalRequest:
PUT
/test01/new_hello.txt

content-md5:5Z/5eUEET4XfUpfhwwLSYA==
host:xxxxxxxxxxxxxxxxxxxxxxx
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230324T221959Z

content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
StringToSign:
AWS4-HMAC-SHA256
20230324T221959Z
20230324/us-east-1/s3/aws4_request
8e31ee98bd6dac6710e9297fc5bec14451d8357cfe9ef9bf648db20dc6c7510f
Signature:
312cedb98b5126ae3872a97193c0062feb37dfba0346830a22dfeb09985776db
Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7fe32fbf1820>
Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://xxxxxxxxxxxxxxxxxxxxxxx/test01/new_hello.txt, headers={'User-Agent': b'Botocore/1.29.76 Python/3.8.10 Linux/5.4.0-126-generic', 'Content-MD5': b'5Z/5eUEET4XfUpfhwwLSYA==', 'Expect': b'100-continue', 'X-Amz-Date': b'20230324T221959Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxxxxxxx/20230324/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=312cedb98b5126ae3872a97193c0062feb37dfba0346830a22dfeb09985776db', 'amz-sdk-invocation-id': b'2afbd511-1788-4a2d-9d4c-e75df23455b5', 'amz-sdk-request': b'attempt=1', 'Content-Length': '12'}>
Certificate path: .venv/lib/python3.8/site-packages/botocore/cacert.pem
Waiting for 100 Continue response.
No response seen from server, continuing to send the response body.
Response headers: {'ETag': '"e59ff97941044f85df5297e1c302d260"', 'Content-Length': '0', 'Server': 'Nutanix Objects', 'x-amz-id-2': '210000+23591+50446830+5986179', 'x-amz-request-id': '210000+23591+50446830+5986179', 'x-ntnx-id': '210000+23591+50446830+5986179', 'Date': 'Fri, 24 Mar 2023 22:20:01 GMT', 'Accept-Ranges': 'bytes'}
Response body:
b''
Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fe32f2d1280>
No retry needed.
Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fe32f2d12e0>>

{'ResponseMetadata': {'RequestId': '210000+23591+50446830+5986179', 'HostId': '210000+23591+50446830+5986179', 'HTTPStatusCode': 200, 'HTTPHeaders': {'etag': '"e59ff97941044f85df5297e1c302d260"', 'content-length': '0', 'server': 'Nutanix Objects', 'x-amz-id-2': '210000+23591+50446830+5986179', 'x-amz-request-id': '210000+23591+50446830+5986179', 'x-ntnx-id': '210000+23591+50446830+5986179', 'date': 'Fri, 24 Mar 2023 22:20:01 GMT', 'accept-ranges': 'bytes'}, 'RetryAttempts': 0}, 'ETag': '"e59ff97941044f85df5297e1c302d260"'}

Current Behavior

This is the detailed log of the script with aiobocotore:

Calculating signature using v4 auth.
CanonicalRequest:
PUT
/test01/new_hello.txt

content-md5:5Z/5eUEET4XfUpfhwwLSYA==
host:xxxxxxxxxxxxxx
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230324T220739Z

content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
StringToSign:
AWS4-HMAC-SHA256
20230324T220739Z
20230324/us-east-1/s3/aws4_request
54cd56ec14d01322d06196d0989dc3000b7e417b036e23f58ce036ca37349faa
Signature:
df00cdb61b75acd16d0d4b8d6c453c448c6cb48fb201506b3c211e09bfcab224
Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7f337e0c94c0>
Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://xxxxxxxxxxxxxx/test01/new_hello.txt, headers={'User-Agent': b'Botocore/1.29.76 Python/3.8.10 Linux/5.4.0-126-generic', 'Content-MD5': b'5Z/5eUEET4XfUpfhwwLSYA==', 'Expect': b'100-continue', 'X-Amz-Date': b'20230324T220739Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxx/20230324/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=df00cdb61b75acd16d0d4b8d6c453c448c6cb48fb201506b3c211e09bfcab224', 'amz-sdk-invocation-id': b'34297524-7fd0-4076-9433-5e01885e7ab7', 'amz-sdk-request': b'attempt=3; max=5', 'Content-Length': '12'}>
Event needs-retry.s3.PutObject: calling handler <aiobotocore.retryhandler.AioRetryHandler object at 0x7f337d297f70>
retry needed, retryable exception caught: Read timeout on endpoint URL: "https://xxxxxxxxxxxxxx/test01/new_hello.txt"
Traceback (most recent call last):
  File ".venv/lib/python3.8/site-packages/aiobotocore/httpsession.py", line 208, in send
    response = await self._session.request(
  File ".venv/lib/python3.8/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File ".venv/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 899, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File ".venv/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerTimeoutError: Timeout on reading data from socket

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv/lib/python3.8/site-packages/aiobotocore/retryhandler.py", line 152, in _should_retry
    return await resolve_awaitable(
  File ".venv/lib/python3.8/site-packages/aiobotocore/_helpers.py", line 15, in resolve_awaitable
    return await obj
  File ".venv/lib/python3.8/site-packages/aiobotocore/retryhandler.py", line 174, in _call
    checker(attempt_number, response, caught_exception)
  File ".venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File ".venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File ".venv/lib/python3.8/site-packages/aiobotocore/endpoint.py", line 181, in _do_get_response
    http_response = await self._send(request)
  File ".venv/lib/python3.8/site-packages/aiobotocore/endpoint.py", line 285, in _send
    return await self.http_session.send(request)
  File ".venv/lib/python3.8/site-packages/aiobotocore/httpsession.py", line 247, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://xxxxxxxxxxxxxx/test01/new_hello.txt"

This will continue for quite some time ... until it reaches 'amz-sdk-request': b'attempt=5; max=5' and breaks.

Pedro Ricardo C Souza · Answer 3 · Sat Mar 25 2023 07:03:58 GMT+0800 (China Standard Time)

About the last 2 tasks that I left unmarked there ...

I don't know how to use aiohttp to ensure that the issue is not there.
The 3.8.x python is currently a requirement, so I didn't test this in new python versions.

Alexander Mohr · Answer 4 · Sat Mar 25 2023 08:44:08 GMT+0800 (China Standard Time)

oo, I think this is related to how aiohttp treats 100 continues by detault (expect100: https://docs.aiohttp.org/en/stable/client_reference.html...will investigate asap. Obrigado!

Alexander Mohr · Answer 5 · Sat Mar 25 2023 08:46:21 GMT+0800 (China Standard Time)

btw could you try older versions of aiobotocore to see if this is an injection, we use for example put_object all the time, but we're not on the latest yet.

Pedro Ricardo C Souza · Answer 6 · Sat Mar 25 2023 10:04:10 GMT+0800 (China Standard Time)

Ok ... sure
I've deleted and re-created the virtual environment several times to test with:

aiobotocore       2.5.0
aiobotocore       2.0.1  
aiobotocore       1.4.2
aiobotocore       1.0.7 
aiobotocore       0.12.0
aiohttp           3.3.1

I could not go down any further on the aiohttp because it breaks aiobotocore, so i kept it at the minimun.

All tests have give the same timeout issue.

Pedro Ricardo C Souza · Answer 7 · Mon Apr 17 2023 22:16:28 GMT+0800 (China Standard Time)

Any news on this subject?

Alexander Mohr · Answer 8 · Tue Apr 18 2023 10:56:11 GMT+0800 (China Standard Time)

thanks for the info, so not injection. so I can't reproduce this because you're using a custom endpoint_url, how can I reproduce?

Alexander Mohr · Answer 9 · Fri Jul 07 2023 13:19:42 GMT+0800 (China Standard Time)

if I don't get a way I can repro this I'll have to close unfortunately as for network level issues like this I need to be able to repro locally.

Pedro Ricardo C Souza · Answer 10 · Fri Jul 07 2023 21:18:06 GMT+0800 (China Standard Time)

I understand ...
I can make tests and post logs here ... but unfortunately I can't give you direct access to this S3 instance that our IT team bought.

But I'll talk to them and see if there is a safe way to enable some sort of temporary and controlled external access.

Pedro Ricardo C Souza · Answer 11 · Sat Nov 25 2023 06:42:25 GMT+0800 (China Standard Time)

I'll close this issue because it was an error at the S3 compatible instance implementation.
They fixed something unknown to me and it works like a charm now.

Fail to put on S3 compatible instance

Reproduction Steps

With aiobocotore

With botocore

Expected Behavior

Current Behavior

With `aiobocotore`

With `botocore`