Multipart Uploads using Checksums

[thomas-icomplai]

Describe the bug

I'm using the s3fs library that makes use of aiobotocore underneath. I'm running the library like this:

fs = s3fs.S3FileSystem(s3_additional_kwargs={'ChecksumAlgorithm': 'SHA256'})
with fs.open(f'some_bucket/some_object', 'w', block_size=10 * 1024 * 1024) as f:
  f.write("hello")

Not supplying the kwargs works fine, yet I'm expecting it to work with the latest ChecksumAlgorithm feature enabled.

I'm receiving several errors from aiobotocore:

Traceback (most recent call last):
  File "Python39\lib\site-packages\aiohttp\client_reqrep.py", line 510, in update_body_from_data
    body = payload.PAYLOAD_REGISTRY.get(body, disposition=None)
  File "Python39\lib\site-packages\aiohttp\payload.py", line 118, in get
    raise LookupError()
aiohttp.payload.LookupError


Traceback (most recent call last):
  File "Python39\lib\site-packages\aiobotocore\httpsession.py", line 178, in send
    response = await self._session.request(
  File "Python39\lib\site-packages\aiohttp\client.py", line 507, in _request
    req = self._request_class(
  File "Python39\lib\site-packages\aiohttp\client_reqrep.py", line 313, in __init__
    self.update_body_from_data(data)
  File "Python39\lib\site-packages\aiohttp\client_reqrep.py", line 512, in update_body_from_data
    body = FormData(body)()
  File "Python39\lib\site-packages\aiohttp\formdata.py", line 37, in __init__
    self.add_fields(*fields)
  File "Python39\lib\site-packages\aiohttp\formdata.py", line 107, in add_fields
    raise TypeError(
TypeError: Only io.IOBase, multidict and (name, file) pairs allowed, use .add_field() for passing more complex parameters, got <botocore.httpchecksum.AwsChunkedWrapper object at 0x000001187CDEC0A0>


Traceback (most recent call last):  
  < my code snipped >
    f.write("hello")
  File "Python39\lib\site-packages\fsspec\spec.py", line 1664, in close
    self.flush(force=True)
  File "Python39\lib\site-packages\fsspec\spec.py", line 1535, in flush
    if self._upload_chunk(final=force) is not False:
  File "Python39\lib\site-packages\s3fs\core.py", line 2095, in _upload_chunk
    self.commit()
  File "Python39\lib\site-packages\s3fs\core.py", line 2110, in commit
    write_result = self._call_s3(
  File "Python39\lib\site-packages\s3fs\core.py", line 1962, in _call_s3
    return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
  File "Python39\lib\site-packages\fsspec\asyn.py", line 86, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "Python39\lib\site-packages\fsspec\asyn.py", line 66, in sync
    raise return_result
  File "Python39\lib\site-packages\fsspec\asyn.py", line 26, in _runner
    result[0] = await coro
  File "Python39\lib\site-packages\s3fs\core.py", line 332, in _call_s3
    return await _error_wrapper(
  File "Python39\lib\site-packages\s3fs\core.py", line 137, in _error_wrapper
    raise err
  File "Python39\lib\site-packages\s3fs\core.py", line 110, in _error_wrapper
    return await func(*args, **kwargs)
  File "Python39\lib\site-packages\aiobotocore\client.py", line 251, in _make_api_call
    http, parsed_response = await self._make_request(
  File "Python39\lib\site-packages\aiobotocore\client.py", line 271, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "Python39\lib\site-packages\aiobotocore\endpoint.py", line 80, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "Python39\lib\site-packages\aiobotocore\endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "Python39\lib\site-packages\aiobotocore\hooks.py", line 50, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "Python39\lib\site-packages\aiobotocore\_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "Python39\lib\site-packages\aiobotocore\retryhandler.py", line 89, in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
  File "Python39\lib\site-packages\aiobotocore\_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "Python39\lib\site-packages\aiobotocore\retryhandler.py", line 107, in _call
    should_retry = await self._should_retry(attempt_number, response,
  File "Python39\lib\site-packages\aiobotocore\retryhandler.py", line 129, in _should_retry
    return await resolve_awaitable(
  File "Python39\lib\site-packages\aiobotocore\_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "Python39\lib\site-packages\aiobotocore\retryhandler.py", line 146, in _call
    checker(attempt_number, response,
  File "Python39\lib\site-packages\botocore\retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "Python39\lib\site-packages\botocore\retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "Python39\lib\site-packages\aiobotocore\endpoint.py", line 149, in _do_get_response
    http_response = await self._send(request)
  File "Python39\lib\site-packages\aiobotocore\endpoint.py", line 229, in _send
    return await self.http_session.send(request)
  File "Python39\lib\site-packages\aiobotocore\httpsession.py", line 220, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: Only io.IOBase, multidict and (name, file) pairs allowed, use .add_field() for passing more complex parameters, got <botocore.httpchecksum.AwsChunkedWrapper object at 0x00000197488C50A0>

Process finished with exit code 1

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.3.4
aiohttp==3.8.1
aioitertools==0.10.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.4.0
azure-core==1.24.2
azure-storage-blob==12.13.0
black==22.6.0
boto3==1.24.27
botocore==1.27.27
certifi==2022.6.15
cffi==1.15.1
charset-normalizer==2.1.0
click==8.1.3
colorama==0.4.5
cryptography==37.0.4
DateTime==4.5
frozenlist==1.3.0
fsspec==2022.5.0
idna==3.3
isodate==0.6.1

Environment:

• Python Version: 3.9
• OS name and version: win11

Additional context
Add any other context about the problem here.

[Alexander Mohr]

repro'd, thanks, will investigate

[Terri Cain]

terricain/aioboto3#265 - Just seen this also

[Terri Cain]

https://github.com/boto/botocore/blob/develop/botocore/httpchecksum.py#L346

So there be the problem, i think if AwsChunkedWrapper was an async iterator, that should suffice, I'm gonna have a crack at it and see where i get