[4.3.1] Missing support for Response.raw.stream() API from urllib3 v2
mgorny opened this issue · comments
I've noticed this because [https://pypi.org/project/dns-lexicon/ dns-lexicon]'s test suite started failing after upgrading to urllib3-2. A trivial reproducer is:
import requests
import vcr
with vcr.use_cassette("test.yaml"):
r = requests.get("https://gentoo.org", stream=True)
print(list(r.raw.stream()))Without vcrpy, this program works. With vcrpy, it throws:
Traceback (most recent call last):
File "/tmp/test.py", line 7, in <module>
print(list(r.raw.stream()))
^^^^^^^^^^^^
AttributeError: 'VCRHTTPResponse' object has no attribute 'stream'
I confirm that this works with urllib3==1.26.16 and does not work with urllib3==2.0.3. Labelling as a bug…
@mgorny I cannot find use of .raw.stream in DNS lexicon code at https://github.com/AnalogJ/lexicon . Is there a log and/or report for the failing code that you ran into?
Yes, it happens through boto3 (or botocore) AFAIR. I'll test the PR in a minute. I may also be able to dig up the details then.
These test failures:
FAILED lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_authenticate - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: 'VCRHTTPResponse' object has no attribute 'stream'
FAILED lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_authenticate_private_zone_false - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: 'VCRHTTPResponse' object has no attribute 'stream'
FAILED lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_for_A_with_valid_name_and_content - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: 'VCRHTTPResponse' object has no attribute 'stream'
FAILED lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_for_CNAME_with_valid_name_and_content - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: 'VCRHTTPResponse' object has no attribute 'stream'
[...]Full log: dev-python:dns-lexicon-3.12.0:20230621-041649.log
I ran out of time right now, so I can test your patch later today.
Ok, I get the same test failures from vcrpy as the CI does with the patch applied, so I guess it's not yet ready to be tested.
@mgorny my impression is that the latest version of the pull request does fix the issue. To reproduce:
cd "$(mktemp -d)"
python3.10 -m venv venv
source venv/bin/activate
git clone --branch issue-714-response-raw-stream-urllib3-v2 https://github.com/kevin1024/vcrpy && ( cd vcrpy && pip3 install -e . )
git clone https://github.com/AnalogJ/lexicon
pip3 install pytest boto3==1.26.156 botocore==1.29.156
pip3 install 'urllib3>=2'
cd lexicon/
git checkout v3.12.0
hash pytest
pytest -v -- lexicon/tests/providers/test_route53.py::Route53ProviderTestsThe pytest call is based on this list of tests failing previously:
# grep "FAILED.*stream" ~/Desktop/dev-python.dns-lexicon-3.12.0.20230621-041649.log | awk '{print $2}' | sed -e 's/\x1b\[[0-9;]*m//g'
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_authenticate
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_authenticate_private_zone_false
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_for_A_with_valid_name_and_content
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_for_CNAME_with_valid_name_and_content
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_for_TXT_with_fqdn_name_and_content
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_for_TXT_with_full_name_and_content
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_for_TXT_with_valid_name_and_content
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_multiple_times_should_create_record_set
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_create_record_with_duplicate_records_should_be_noop
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_delete_record_by_filter_should_remove_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_delete_record_by_filter_with_fqdn_name_should_remove_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_delete_record_by_filter_with_full_name_should_remove_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_delete_record_by_identifier_should_remove_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_delete_record_with_record_set_by_content_should_leave_others_untouched
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_delete_record_with_record_set_name_remove_all
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_list_records_after_setting_ttl
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_list_records_should_handle_record_sets
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_list_records_with_fqdn_name_filter_should_return_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_list_records_with_full_name_filter_should_return_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_list_records_with_invalid_filter_should_be_empty_list
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_list_records_with_name_filter_should_return_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_list_records_with_no_arguments_should_list_all
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_update_record_should_modify_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_update_record_should_modify_record_name_specified
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_update_record_with_fqdn_name_should_modify_record
lexicon/tests/providers/test_route53.py::Route53ProviderTests::test_provider_when_calling_update_record_with_full_name_should_modify_recordWhat do you think?
Yes, I can confirm that the newest version works. Thanks!