When uploading, checking 'Expect' header results in "TypeError: sequence item 0: expected str instance, bytes found"
genman opened this issue · comments
Describe the bug
Uploading a binary file to S3 instance (not AWS). This error is occurring:
Traceback (most recent call last):
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/httpsession.py", line 464, in send
urllib_response = conn.urlopen(
File "/miniconda/envs/iris/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/miniconda/envs/iris/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/awsrequest.py", line 91, in request
if headers.get('Expect', b'') == b'100-continue':
^^^ Why is this b''?
File "/miniconda/envs/iris/lib/python3.10/_collections_abc.py", line 824, in get
return self[key]
File "/miniconda/envs/iris/lib/python3.10/site-packages/urllib3/_collections.py", line 261, in getitem
return ", ".join(val[1:])
TypeError: sequence item 0: expected str instance, bytes found
^^^ I'm not sure the value of 'val' but
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
...
File "/miniconda/envs/iris/lib/python3.10/site-packages/boto3/s3/inject.py", line 145, in upload_file
return transfer.upload_file(
File "/miniconda/envs/iris/lib/python3.10/site-packages/boto3/s3/transfer.py", line 371, in upload_file
future.result()
File "/miniconda/envs/iris/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/miniconda/envs/iris/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
raise self._exception
File "/miniconda/envs/iris/lib/python3.10/site-packages/s3transfer/tasks.py", line 139, in call
return self._execute_main(kwargs)
File "/miniconda/envs/iris/lib/python3.10/site-packages/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/miniconda/envs/iris/lib/python3.10/site-packages/s3transfer/upload.py", line 764, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/client.py", line 989, in _make_api_call
http, parsed_response = self._make_request(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/client.py", line 1015, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/endpoint.py", line 119, in make_request
return self._send_request(request_dict, operation_model)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/endpoint.py", line 202, in _send_request
while self._needs_retry(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/endpoint.py", line 354, in _needs_retry
responses = self._event_emitter.emit(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
return self._emit(event_name, kwargs)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
response = handler(**kwargs)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/retryhandler.py", line 207, in call
if self._checker(**checker_kwargs):
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/retryhandler.py", line 284, in call
should_retry = self._should_retry(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/retryhandler.py", line 307, in _should_retry
return self._checker(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/retryhandler.py", line 363, in call
checker_response = checker(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/retryhandler.py", line 247, in call
return self._check_caught_exception(
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
raise caught_exception
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/endpoint.py", line 281, in _do_get_response
http_response = self._send(request)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/endpoint.py", line 377, in _send
return self.http_session.send(request)
File "/miniconda/envs/iris/lib/python3.10/site-packages/botocore/httpsession.py", line 509, in send
raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes found
Expected Behavior
Upload to work, no exception
Current Behavior
Exception as shown
Reproduction Steps
Likely requires S3 server that's not AWS
Possible Solution
Why are headers checked like this? Maybe use str types not byte?
Additional Information/Context
No response
SDK version used
1.34.24
Environment details (OS name and version, etc.)
Linux, Python 3.10
We hit this one too. Solution was to revert to an older version. This shouldn't have made it out to release.
We hit this one too. 1.34.31
Hi everyone, thanks for reaching out. It's not immediately clear why you're seeing issues here, this code has been in Botocore for ~10 years (ref). The reason we fall back to an empty byte string for comparison is header values should always be bytes, that's the existing typing contract.
We'll need some more information to investigate further. If you can answer these questions, it will help determine what's happening.
- Did everyone start seeing this recently? If so, can you provide the version you started seeing this issue?
- In the cases people saw it stop with a rollback, what version is stable?
- Is everyone using a non-AWS endpoint for communicating with an s3 emulator?
- What version of Python and urllib3 are you using?
Using the latest version here (installed via pymongo[aws] so I don't control the version).
Was working fine up until yesterday and today I also ran into it.
One interesting observation: Because I'm curious what response caused it to break, I launched Proxyman which is a packet inspection tool, but before I went and checked the packet, I noticed the error went away. I tried closing Proxyman, and the error occurred. Launching it again, and I was able to successfully deploy my code without even changing the library versions.
I am facing the same issue using AWS SAM version 1.108.0 and Python 3.11. When running sam package/deploy on one of the stock SAM template I get:
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1mk1hlsoyqzcb
A different default S3 bucket can be set in samconfig.toml
Or by specifying --s3-bucket explicitly.
Error: Unable to upload artifact hello_world referenced by CodeUri parameter of HelloWorldFunction resource.
An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes found
I'm running into this issue as well, seemingly at random when using the R kernels in a Sagemaker Notebook Instance to send a file to S3 using the CLI on AWS.
We don't control the versioning of the base AWS Sagemaker kernels but we're using Amazon Linux 2, Jupyter Lab 3
(notebook-al2-v2)
I used ```conda list```` in an instance that failed today and am sharing
Did everyone start seeing this recently? If so, can you provide the version you started seeing this issue?
This issue started happening very recently, we noticed it yesterday (2024-02-02) It's listing botocore as using 1.34.32
In the cases people saw it stop with a rollback, what version is stable?
Have not attempted.
Is everyone using a non-AWS endpoint for communicating with an s3 emulator?
I believe I'm on an AWS endpoint, as I'm running this in a notebook instance in Sagemaker.
What version of Python and urllib3 are you using?
Python is listed as 3.11.5, but I'm accessing the CLI from the system() command in R.
URLlib is 1.26.18
I had been digging into if this was a CLI change but a coworker found this issue and the error messages match.
i noticed this happens when using urllib 2.* instead of 1.26.18 for me with boto3 1.34.33
I can confirm that I have the same issue and that running Proxyman "solves it". Could it be some sort of certificate issue somewhere since running the call through Proxyman works?
same issue here
but upgrade to 1.34.33 solved it
For anyone who's recently been able to produce this and upgrading fixed it, can you try installing urllib3==2.2.0? The only lead we have currently is their recent release on 2024-01-31.
We do not officially support this version yet, but for users upgrading all dependencies to the latest version, they may end up with dependencies outside of the supported range.
For anyone who's recently been able to produce this and upgrading fixed it, can you try installing urllib3==2.2.0? The only lead we have currently is their recent release on 2024-01-31.
We do not officially support this version yet, but for users upgrading all dependencies to the latest version, they may end up with dependencies outside of the supported range.
I'm having this issue right now. tried installing urllib3=2.2.0. But, the issue is still there :(
@gihanw try installing urllib3<2.2.0. We're trying to test if the issue is in the latest urllib3 release.
@gihanw try installing urllib3<2.2.0. We're trying to test if the issue is in the latest urllib3 release.
I was already on 1.26.18. But, it didn't work.
Finally this worked. aws/aws-sam-cli#6629 (comment)
@nateprewitt , the previous installation was via brew and was the same version.
SAM CLI, version 1.108.0
We have multiple stacks and it worked for some stacks. But, not this particular one. if that information is going to help.
Same issue here,
Downgrading botocore to 1.34.33 solved it. urlib3 was below version 2.2.0 and it was still broken.
Uninstalling Homebrew SAM and using the real installer fixed this for me.
install urllib3==2.0.7 fixed mine.
rest of the packages are still the latest.
I am using docker image built by cog. Running this docker image in a GCP VM. Tried multiple solutions above but couldn't fix it. Happy to connect to over discord to resolve this quickly. https://discord.gg/4NyDaSdx
install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.
This worked for me as well
Ran sam deploy to deploy a Lambda function. Received error:
error: an http client raised an unhandled exception: sequence item 0: expected str instance, bytes found
Doing #3111 (comment) fixed the issue for me too.
Uninstalling Homebrew SAM and using the real installer fixed this for me.
Hi are facing the issue when boto3 1.34.34 and urllib3 1.26.18. Any quick fix?
I am using docker image built by cog. Running this docker image in a GCP VM. Tried multiple solutions above but couldn't fix it. Happy to connect to over discord to resolve this quickly. https://discord.gg/4NyDaSdx
consider adding something like the following lines to your cog.yaml:
run:
- pip3 install boto3==1.33.11
- pip3 install urllib3==1.26.17
for some reason, specifying the same versions in the python_packages section didn't resolve the issue for me. It seems to be related to a sub-dependency problem.
I get this issue when using AWS Lambda (Python 3.12).
install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.
Could you please tell what is the version of botocore?
install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.
Could you please tell what is the version of botocore?
For me, not botocore, but boto3.
boto3==1.34.31
Hit this a few days ago in a conda environment:
boto3 1.34.34 pyhd8ed1ab_0 conda-forge
botocore 1.34.34 pyhd8ed1ab_0 conda-forge
urllib3 2.2.0 pypi_0 pypi
python 3.8.18 hd12c33a_0_cpython conda-forge
Cannot work out why it happens, but if I delete and rebuild the conda environment seems to be ok for a while.
Update: Noticed from coping and pasting above that urllib was coming from pypi, issue was an incompatible urllib version was being installed by mypy with "--install-types" argument. Manually installing type packages and keeping correct urllib version solved the issue.
We are also facing the same issue. I think it is not due to botocore, but because urllib3 updated from using bytes to str
I also ran into the same issue. urllib3==2.0.7 seems to fixes it but have another sagemaker dependency conflict. Is this related to the CVE-2024-2626 security updates? https://aws.amazon.com/security/security-bulletins/AWS-2024-001/
This worked:
!pip install urllib3==2.0.7 --force-reinstall
Throws this warning in SageMaker Studio thought: which is weird in general
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
distributed 2022.7.0 requires tornado<6.2,>=6.0.3, but you have tornado 6.4 which is incompatible.
sagemaker 2.199.0 requires urllib3<1.27, but you have urllib3 2.0.7 which is incompatible.
Ran into the same issue, force installed boto3==1.34.33 , which reduced urllib3 to 2.0.7, and it worked
Just ran into this issue using botocore 1.34.35. Downgrading urllib3 from version 2.2.0 to 2.1.0 fixed the issue.
install urllib3==2.0.7 fixed mine. rest of the packages are still the latest.
This solution has worked for me.
I was running an AWS Sagemaker MLOps workshop notebook on a Data Science 3.0 on Sagemaker Classic notebook and ran into the same issue.
Python version = 3.10
boto3 version = 1.34.33
Sagemaker version = 2.206.0
Then Sagemaker version = 2.207.1 has fixed the issue.
Ran into the same issue, force installed boto3==1.34.33 , which reduced urllib3 to 2.0.7, and it worked
How come? boto3 has conflict with urllib3 >= 2.0
Just ran into this issue using botocore 1.34.35. Downgrading urllib3 from version 2.2.0 to 2.1.0 fixed the issue.
How is this possible.. boto3 (even the latest version) has conflict with urllib3 >= 2.0
Seems like Gentoo removes this setup.py version check: https://github.com/gentoo/gentoo/tree/master/dev-python/botocore
# unpin deps
sed -i -e "s:>=.*':':" setup.py || die
I am also facing same issue: botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes found
Could anyone help me with exact version of boto3 , botocore and urllib3 to use in order to overcome this issue.
We were able to resolve the issue by pinning the following versions:
urllib3==1.26.13
botocore==1.29.165
boto3==1.26.93
This is in a python3.11-buster Docker image.
We were able to solve this issue by having the following packages:
boto3==1.34.33
botocore==1.34.40
urllib3==1.26.18
for python 3.8.16 debian based docker image.
Also getting this error with Lambda Python 3.12.
What's the best way to solve this in Lambda? Create custom layer with another version of boto3?
Hi everyone,
Coming back to our previous responses, we've confirmed this issue is specific to urllib3 2.2.0. We do not support this version of urllib3 and specifically have botocore pinned to urllib3<2.1 due to breaking changes made in that minor version release.
We've confirmed that while the upload_file workflow works as expected with urllib3 2.1.0, it broke with the release of 2.2.0 due to what appeared to be an innocuous type change. Specifically on this line, all headers are now getting coverted to urllib3's HTTPHeaderDict which wasn't the case before. This has implicit requirements that all headers have string values, while previously urllib3 has supported bytes or headers (Requests and botocore prefer bytes as they are more correct when serializing over the wire).
This is the current root cause of the breakage, I've opened a ticket (urllib3/urllib3#3343) in the urllib3 tracker to discuss options. For now, installing any version of boto3 normally will work as expected. The only way to hit this case it to force a urllib3 upgrade beyond botocore's defined support range, or install another project that explicitly requires urllib3>=2.2.0.
For the time being the remediation step is to ensure you are using this in your installation requirements. There is no need to pin boto3 or botocore versions:
urllib3<2.2
If anyone can help identify how urllib3 2.2.0 go into their dependency closure, that may be helpful. A common way users break their installations is using --upgrade independently of installing their dependencies.
Also getting this error with Lambda Python 3.12. What's the best way to solve this in Lambda? Create custom layer with another version of boto3?
Hello @RomainBsb, you need to make sure your Lambda layers do not use urllib3==2.2.0. For example, if you made a layer for requests, requests will use latest 2.2.0 as a dependency. As a result, this will cause your lambda to run into issues with botocore.
To make sure, you don't encounter any issues you can install requests with the --no-dependencies flag and its dependencies manually while making sure urllib3<2.2.0.
For example,
if [ "$pkg" == "requests" ]; then
# Force urllib3 to version 2.1.0, as 2.2.0 causes issues with botocore
pip install --platform manylinux2014_x86_64 \
--target="$pkg_dir/python" \
--implementation cp \
--python-version $python_version \
--only-binary=:all: --upgrade \
--no-dependencies \
requests
pip install --platform manylinux2014_x86_64 \
--target="$pkg_dir/python" \
--implementation cp \
--python-version $python_version \
--only-binary=:all: --upgrade \
urllib3==2.1.0
pip install --platform manylinux2014_x86_64 \
--target="$pkg_dir/python" \
--implementation cp \
--python-version $python_version \
--only-binary=:all: --upgrade \
idna
pip install --platform manylinux2014_x86_64 \
--target="$pkg_dir/python" \
--implementation cp \
--python-version $python_version \
--only-binary=:all: --upgrade \
charset-normalizer
pip install --platform manylinux2014_x86_64 \
--target="$pkg_dir/python" \
--implementation cp \
--python-version $python_version \
--only-binary=:all: --upgrade \
certifi
fi
Hello! urlllib3 maintainer here. Sorry for the hassle.
urllib3 2.2.1 should have fixed this issue.
Thanks for the release and checking back in @pquentin! With 2.2.1 out, it shouldn't be possible to hit this case when force upgrading urllib3 outside of the supported range. I'll go ahead and resolve this as completed.
For anyone hitting this in the future, you'll need to ensure you are not installing urllib3 2.2.0. All other versions released as of Feb 20, 2024 should otherwise work in this case.
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
@narenderAwen It seems setup.py is limiting urllib3>=1.25.4,<2.1, so it won't install urllib3 2.2.1. Is this intended?
so we just hit this issue deploying and doing final testing -
from my observations (and being bitten by requests before) - just use urllib3 instead of requests python layer.
urllib3 comes out of the box with botocore.
if it's a burden to rewrite - you can just throw in the code to chatgpt /claude opus and ask it to replace it for you.
sample urllib3 rewrite from requests layer.
side note - you may consider using docker to build layers /
just throw a requirements.txt file in layer directory
(it avoids problems like uploading mac m1/m3 arm32 layers to prod)
build_via_docker.sh
# Set the Python version
PYTHON_VERSION="3.12"
echo "Cleaning aws-layer and lambda_function.zip..."
rm -rf aws-layer
rm -f lambda_function.zip
# Create the necessary directories for the Python version
mkdir -p "aws-layer/python/lib/python$PYTHON_VERSION/site-packages"
# Use Docker to install packages according to the requirements.txt in the specified directory
sudo docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python$PYTHON_VERSION:latest" /bin/sh -c \
"pip install -r requirements.txt -t aws-layer/python/lib/python$PYTHON_VERSION/site-packages/; exit"
# Create a zip file of the aws-layer
cd aws-layer
zip -r ../lambda_function.zip .
cd ..
# Clean up by removing the aws-layer directory
rm -rf aws-layer