[Duplicate of #719?] Upgrading to VCR.py that uses urllib3 v2+ causes string decoding problems with requests

Question

[Duplicate of #719?] Upgrading to VCR.py that uses urllib3 v2+ causes string decoding problems with requests

andkon opened this issue 7 months ago · comments

I've been on VCR.py 4.2.0, and am upgrading to 5.1.0. I'm also upgrading to python 3.10, and for various reasons, I have to upgrade urllib3 to v2.

When I run a cassette using urllib3 v2+ and a vcrpy version above 4.2.1, it causes the body of a cassette-saved response to return a string but without decoding the characters from utf-8.

Here's how the body of one requests response looks, in python 3.8.12 / vcr 4.2.0 / urllib3 1.26.7:

$ response.text
'{"alcoholPercentage":40.0,"basicPrice":55.9,"itemNumber":"492884","kosher":false,"listDelistCode":"2","producer":"Jas. Hennessy & Co.","releaseDate":"","sccNumber":"10081753827790","upcNumber":"081753825492","vqa":false,"itemName":"Hennessy VSOP","producingCo....}'
$ type(response.text)
<class 'str'>

And in 3.10.13, with vcr.py 5.1.0 and urllib3 2.1.0 (though I've tested it with everything above 4.2.1.):

$ response.text
'\x1f�\x08\x00\x00\x00\x00\x00\x00\x03�U�r�6\x0c�\x15��ӓ-�N�8��am��v���r�c\x07� �\tEjIȶ��\x7f/()����B�\x1f\x00�\x01x��\n@��4j�6EMP`p\x7f=\t\'� \x01\'ӭ�)#�yx7\n$a��T\t��>���-\x16��(x3��H\x0e��(P��\x1a��2\x19�\x063���ɚ���\r\\(>���\\+~\x14+\x13��E��p\r�C\x18pizJ5�L\x16����bv{{7acS���9\x13b��+��t|��7�\x12�ě퉐���4�l�.?[�\\��\x0b\x16�h6�L�!�4�Mb߭�-*��\x16ҷ��Ab����\x1d\x10ɧ^�K���x�l p�!O��1�A$\nu&L.�\x1e���\x08��8�p��(,�Fy\x7f!�\x00Q�����q\x02�5Z�L]��·\x11��I�\x11T��K\x1b�\x03;G�\x0e}\x0eo��\x01g���|\x10\x1a���D\x02�7�{:��\x0b*\x050;W\x19C�/G\x1c$c%(U�}s�s\x0fZ*\x05#�*.�\x7f���4G]hʇ�L䶑$*T�\'E�@�bEn,�����\t�9\x11�*�\n�A�2|���Aj|\x01�\x12�3\x12S�|�?c&���\x15���F�[�\\\x12��>�\x0e�C��Ą���r\x10�x�L=��\x1d�#c��_|u;\x17գ\x1fkS�]\x19M��>�"�zi\x14�ag\x06��XK\x0bt�J��5��I\x1a\\\'�W\x00�|�>�=ݩ{�\x7fA\x1e���\x1c�M�O�\x1e�͆6Tp\\���\x064Ib\x15���Ɯ���\x19���m_����q�˦����l�-����][㯙����O��\x13�X��Hr\x0c�����m���{��ذv��� ��\\Y�5.Y]�q\n��\x17�\x10�\x12��&܆�|�\x13\x7f\x17��=\x7f��e��\x17�w4e�+���\x03V�ɧ�\x14�\x0c�ȃ�0r\x17]Gw�,ZD�\x1a�\u05fa\x08��Ά����?^k,.r��\x7fL4�-&�1$�\x07m� n���\x11\u0603T�HՍ5Xnv\x1f\x19�y\x131[z�q��0��Hkh����K��d!/&��a³<Xo��M\x1e�I�v%??���:\\����t�]�X��!��\x15?��NV���oD�\x1b\x13?6^Ǐ�?��w�է��\x7f\x00\x00\x00��\x03\x00t\x1c~\x14�\x06\x00\x00'
$ type(response.text)
<class 'str'>

And here's the cassette it's playing back:

interactions:
- request:
    body: scope=api%3A%2F%2F0d1d8f09-6283-47c6-81d0-91c48f04574b%2F.default&grant_type=client_credentials&client_id=962a480f-825b-4cc7-a86c-1477d0f33eaf&client_secret=Cxq7Q~iOLxrU~iHo6s6egr2~43UonP9d8dWYr
    headers:
      Accept:
      - '*/*'
      Accept-Encoding:
      - gzip
      Accept-Language:
      - en-ca
      Connection:
      - keep-alive
      Content-Length:
      - '194'
      Content-Type:
      - application/x-www-form-urlencoded; charset=utf-8
      Cookie:
      - fpc=AmVgYCVQ2EFHo3L0ClzRivc; stsservicecookie=estsfd; x-ms-gateway-slice=estsfd
      Host:
      - login.microsoftonline.com
      User-Agent:
      - LCBO/356 CFNetwork/1329 Darwin/21.3.0
    method: POST
    uri: https://login.microsoftonline.com/4a5b7942-b0b1-4244-b601-19f8e6c33e48/oauth2/v2.0/token
  response:
    body:
      string: '{"token_type":"Bearer","expires_in":3599,"ext_expires_in":3599,"access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImpTMVhvMU9XRGpfNTJ2YndHTmd2UU8yVnpNYyIsImtpZCI6ImpTMVhvMU9XRGpfNTJ2YndHTmd2UU8yVnpNYyJ9.eyJhdWQiOiJhcGk6Ly8wZDFkOGYwOS02MjgzLTQ3YzYtODFkMC05MWM0OGYwNDU3NGIiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC80YTViNzk0Mi1iMGIxLTQyNDQtYjYwMS0xOWY4ZTZjMzNlNDgvIiwiaWF0IjoxNjQ5NjI3ODA0LCJuYmYiOjE2NDk2Mjc4MDQsImV4cCI6MTY0OTYzMTcwNCwiYWlvIjoiRTJaZ1lHaHVDWGwydHVIRWdSL1BPamkzL1Rza0FRQT0iLCJhcHBpZCI6Ijk2MmE0ODBmLTgyNWItNGNjNy1hODZjLTE0NzdkMGYzM2VhZiIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzRhNWI3OTQyLWIwYjEtNDI0NC1iNjAxLTE5ZjhlNmMzM2U0OC8iLCJvaWQiOiIzYzhlNTJmZS1kOTUxLTRjNzQtYWFjNi03OTY1OTU1YjIzY2IiLCJyaCI6IjAuQVRnQVFubGJTckd3UkVLMkFSbjQ1c00tU0FtUEhRMkRZc1pIZ2RDUnhJOEVWMHM0QUFBLiIsInN1YiI6IjNjOGU1MmZlLWQ5NTEtNGM3NC1hYWM2LTc5NjU5NTViMjNjYiIsInRpZCI6IjRhNWI3OTQyLWIwYjEtNDI0NC1iNjAxLTE5ZjhlNmMzM2U0OCIsInV0aSI6IjRBdzl6anVGSFVXZjFFd3pYajlGQUEiLCJ2ZXIiOiIxLjAifQ.iiTDR4RFCDMeLtokc_B4lML2YnFkFiWMps-O3TsEBecDMrDsdng7xelTma7Z_0f4OfS4mMGMfJ5eNeVAa3RQZMMyAizCedGz-1OD7HbNY2n31GVJIT4TzdLmNL1IH1a1OpHx6Tn_IiTJU7xf8U6bnZNgYOQmCh9HdHwAcpz0VWNE2S0R_R3FV1PBOK49MPDUs80XCqxVWnSl-AWNA_-sWQOxa7K7NNQQ1W_SldGWtN6ekCrqLA2WAT2vi-TIFY-K-t9-xHLnb919AQCkodwNO9URmc8j6Mu_rrGkmDF5TctlzFsiBsL7ZLTTN58qtXPvoO4WjFTO3UHCYsqnJ1vg1Q"}'
    headers:
      Cache-Control:
      - no-store, no-cache
      Content-Length:
      - '1339'
      Content-Type:
      - application/json; charset=utf-8
      Date:
      - Sun, 10 Apr 2022 22:01:43 GMT
      Expires:
      - '-1'
      P3P:
      - CP="DSP CUR OTPi IND OTRi ONL FIN"
      Pragma:
      - no-cache
      Set-Cookie:
      - fpc=AmVgYCVQ2EFHo3L0ClzRiveSJJ2pAQAAAMdK5dkOAAAA; expires=Tue, 10-May-2022
        22:01:44 GMT; path=/; secure; HttpOnly; SameSite=None
      - x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
      - stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
      Strict-Transport-Security:
      - max-age=31536000; includeSubDomains
      X-Content-Type-Options:
      - nosniff
      X-XSS-Protection:
      - '0'
      x-ms-ests-server:
      - 2.1.12621.9 - WUS2 ProdSlices
      x-ms-request-id:
      - ce3d0ce0-853b-451d-9fd4-4c335e3f4500
    status:
      code: 200
      message: OK
- request:
    body: null
    headers:
      Accept:
      - '*/*'
      Accept-Encoding:
      - gzip
      Accept-Language:
      - en-CA;q=1.0, fr-CA;q=0.9
      Authorization:
      - Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImpTMVhvMU9XRGpfNTJ2YndHTmd2UU8yVnpNYyIsImtpZCI6ImpTMVhvMU9XRGpfNTJ2YndHTmd2UU8yVnpNYyJ9.eyJhdWQiOiJhcGk6Ly8wZDFkOGYwOS02MjgzLTQ3YzYtODFkMC05MWM0OGYwNDU3NGIiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC80YTViNzk0Mi1iMGIxLTQyNDQtYjYwMS0xOWY4ZTZjMzNlNDgvIiwiaWF0IjoxNjQ5NjI3ODA0LCJuYmYiOjE2NDk2Mjc4MDQsImV4cCI6MTY0OTYzMTcwNCwiYWlvIjoiRTJaZ1lHaHVDWGwydHVIRWdSL1BPamkzL1Rza0FRQT0iLCJhcHBpZCI6Ijk2MmE0ODBmLTgyNWItNGNjNy1hODZjLTE0NzdkMGYzM2VhZiIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzRhNWI3OTQyLWIwYjEtNDI0NC1iNjAxLTE5ZjhlNmMzM2U0OC8iLCJvaWQiOiIzYzhlNTJmZS1kOTUxLTRjNzQtYWFjNi03OTY1OTU1YjIzY2IiLCJyaCI6IjAuQVRnQVFubGJTckd3UkVLMkFSbjQ1c00tU0FtUEhRMkRZc1pIZ2RDUnhJOEVWMHM0QUFBLiIsInN1YiI6IjNjOGU1MmZlLWQ5NTEtNGM3NC1hYWM2LTc5NjU5NTViMjNjYiIsInRpZCI6IjRhNWI3OTQyLWIwYjEtNDI0NC1iNjAxLTE5ZjhlNmMzM2U0OCIsInV0aSI6IjRBdzl6anVGSFVXZjFFd3pYajlGQUEiLCJ2ZXIiOiIxLjAifQ.iiTDR4RFCDMeLtokc_B4lML2YnFkFiWMps-O3TsEBecDMrDsdng7xelTma7Z_0f4OfS4mMGMfJ5eNeVAa3RQZMMyAizCedGz-1OD7HbNY2n31GVJIT4TzdLmNL1IH1a1OpHx6Tn_IiTJU7xf8U6bnZNgYOQmCh9HdHwAcpz0VWNE2S0R_R3FV1PBOK49MPDUs80XCqxVWnSl-AWNA_-sWQOxa7K7NNQQ1W_SldGWtN6ekCrqLA2WAT2vi-TIFY-K-t9-xHLnb919AQCkodwNO9URmc8j6Mu_rrGkmDF5TctlzFsiBsL7ZLTTN58qtXPvoO4WjFTO3UHCYsqnJ1vg1Q
      Cache-Control:
      - no-cache
      Connection:
      - keep-alive
      Host:
      - api-magento.lcbo.com
      Language:
      - en
      Pragma:
      - no-cache
      User-Agent:
      - LCBO/356 CFNetwork/1329 Darwin/21.3.0
    method: GET
    uri: https://api-magento.lcbo.com/v8/products/492884/
  response:
    body:
      string: !!binary |
        H4sIAAAAAAAAA6xVwXLjNgz9FY6m05Mt2U6cOJnpYW2n3XazsbNy02MHkiCJCUVqSci2ttN/Lygp
        tru99kKPHwDiAXig/gpApaY0aos2RU1QYHB/PQknoyABJ9OtlSkj83l4NwokYfXUVAna4D64vpst
        FtfBKHgzrvRIDsrhKFDS0Rr9uTIZhwYz9qmtyZq0i/sNXCg+otboXCt+FCsTsoNFheBwDeRDGHBp
        eko1nUwW09v51WJ2e3s3YWNTn43vpjkTYtP+K5yodHyh8jeeEr7Em+2JkNTFyjSabMsuP1vQXOuF
        7QsW0mg2rUyhIb00xU1i362eLSrF6BbSt66DQWKIlL/MHRDJp16jS62syXjSbCBwxCFPhtAx8kEk
        CnUmTC7MHq24mQiE5jjOcLyXKCy6Rnl/IbUAUYKtjJamcQKPNVqJTF2M2cK3EZesSZARVKLgSxvr
        AztHkg59Dm+pmAFn6vKi5al8EBqosaBEAso3w3s6sqgLKgUwO1cZQ6UvRxwkYyUoVYF9c95zD1oq
        BSORKi6Bf6TWwDRHXWjKh8RM5LaRJCpUmSdFpkDmYkVuLLOvpM7GCfM5Ee4q8wqpQfoyfK/430Fq
        fAG2EqgzElOrfPc/YyabisUV++6f5ka/W+9cEtXuPooOh0Oo0sSEqamichDIeO9MPe7FHZ0jY/nN
        X3x1OxfVox9rU4BdGU3c6T6/IsN6aRTYYWcG8OFYSwt0kkqvjDXWxkkaXCfhlFcA/Hy5Pt493al7
        k39BHv6hxxyCTctP2B6MzYY2VHBcgaXnBjRJYhWP+aLGnLaU/xnNysRtX8bZ8YbJce3LpoXE94xs
        0y3v18bYXVvjr5nv4+vzT76KE8pYzOVIcgynvKyFsW3n+rR78K7v2LB2/eLwIJa8XFnbNS5ZXcZx
        Cr+2F/AQ+hLG4SbchuK0fOwTfxe92T1/+49luOAXmXc0ZcUr+e/JA1bnyaf9FKMMqsiD0TByF11H
        d9EsWkS9GsLXugip5M6Gs8n1oj9eaywucuy8+X9MNJ0tJv0xJKoHbf0gbq7C6ZwR2INUkEjVjTVY
        bnYfGc15EzFbeuRxtdwwovFIa2j908zvS/ugZCEvJu/4YcKzPFhvpqFNHpNJ33YlPz/8jcg6XJnD
        9+h01l3AWK+RIaWSFT/B2U5WuMnzi29E5xsTPzZex4+bP/6Md5vVp+DvfwAAAP//AwB0HH4UjwYA
        AA==
    headers:
      Allow:
      - OPTIONS, GET
      Connection:
      - keep-alive
      Content-Encoding:
      - gzip
      Content-Language:
      - en
      Content-Type:
      - application/json
      Date:
      - Sun, 10 Apr 2022 22:01:46 GMT
      Referrer-Policy:
      - same-origin
      Server:
      - ZENEDGE
      Strict-Transport-Security:
      - max-age=31536000; includeSubDomains
      Transfer-Encoding:
      - chunked
      Vary:
      - Accept, Cookie, Origin, Accept-Language
      Via:
      - kong/2.5.0
      X-CORRELATION-ID:
      - 61b3f0f6-2fef-45be-9028-c3e0e91abec3
      X-Cache-Status:
      - NOTCACHED
      X-Cdn:
      - Served-By-Zenedge
      X-Content-Type-Options:
      - nosniff
      X-Frame-Options:
      - DENY
      X-Kong-Proxy-Latency:
      - '1'
      X-Kong-Upstream-Latency:
      - '1528'
      X-Zen-Fury:
      - cd87fa3b2c333364c73d6f4b65be340c703b60bb
    status:
      code: 200
      message: OK
version: 1

Thanks for your work! Happy to try to solve this one and put it in a PR, any help in pointing it in the right direction would be appreciated.

Sebastian Pipping · Answer 1 · Sat Dec 09 2023 06:27:21 GMT+0800 (China Standard Time)

@andkon I think this is the same as #719 (which in itself is a feature or a bug depending on view).

Andrew Konoff · Answer 2 · Fri Dec 15 2023 02:12:19 GMT+0800 (China Standard Time)

Thanks @hartwork. Agreed, I think this is a dupe of 719. Closing now.