Duplicate requests in latest_requests() if there are chunks

Question

Duplicate requests in latest_requests() if there are chunks

VelorumS opened this issue 3 years ago · comments

Was fine with 1.0.5, does duplicates in 1.1.0.

The change: b6161ce#r50812634

-            cls.latest_requests[-1] = request
+            pos = cls.latest_requests.index(request)
+            cls.latest_requests[pos] = request

httpretty.historify_request() is called multiple times for the same request.

From here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1081, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 683, in sendall

And here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 699, in sendall

Adrian · Answer 1 · Wed May 19 2021 20:24:05 GMT+0800 (China Standard Time)

These duplicate requests broke my CI. Pinning httpretty<1.1 is a workaround but a fix would be appreciated.

Jason Weir · Answer 2 · Thu May 20 2021 04:49:35 GMT+0800 (China Standard Time)

I also came looking for this because of a broken CI.

Gabriel Falcão · Answer 3 · Thu May 20 2021 05:57:38 GMT+0800 (China Standard Time)

Oh gee, thanks @ChipmunkV for the solution. I'll get a new release out soon and will ping you folks once it's done.

Gabriel Falcão · Answer 4 · Fri May 21 2021 02:09:39 GMT+0800 (China Standard Time)

@ChipmunkV, @Gidgidonihah , @ThiefMaster I'm attempting to write a functional test to reproduce the bug.
Would it be possible for you to share some example code to reproduce the issue?

Jason Weir · Answer 5 · Tue May 25 2021 07:05:13 GMT+0800 (China Standard Time)

FYI, I cloned master after merge and tested it, and my tests are still failing with the wrong number of requests in latest_requests. I don't yet have a MWE example, but I would love to get one worked up.

Gabriel Falcão · Answer 6 · Tue May 25 2021 23:09:33 GMT+0800 (China Standard Time)

@Gidgidonihah I'd love to work with you in a solution for this, I just want to make sure there is an automated test to reproduce the issue.

I tried running the tests from indico-plugins-cern that @ThiefMaster mentioned but ran into some issues and ended up spending too much time, so I gave up on running the test.
I also had trouble understanding what these tests are really trying to accomplish: https://github.com/indico/indico-plugins-cern/blob/03acd644ab2476b37c693250f4ea62b9f26985c1/ravem/tests/operations/connect_test.py#L62-L70

My main concern with httpretty.latest_requests() is whether it have been storing requests incorrectly up to the release 1.0.5. I that is the case, then I'm afraid that you, @ThiefMaster and anyone else who relies on len(httpretty.latest_requests()) might have been relying on an HTTPretty bug rather than a feature.

I take automated testing very seriously and want to make sure that whatever the case is, we can find the best possible solution for HTTPretty and ensure that your tests pass correctly (without false-positives).

How do you think I should proceed with this issue?

Gabriel Falcão · Answer 7 · Tue May 25 2021 23:12:08 GMT+0800 (China Standard Time)

The issue title suggests that the problem occurs "if there are chunks". I suppose this is a problem with chunked requests, is that the case for everyone here?
Would any of you help me reproducing the issue?

Adrian · Answer 8 · Tue May 25 2021 23:15:28 GMT+0800 (China Standard Time)

Feel free to send a PR against my repo that reverts this commit and installs httpretty from git.

That way you don't need to setup the somewhat complex environment locally but still see if the error happens or not.

Alternatively, if you point me to the branch from which I can install the (potentially) fixed version I can give it a try locally.

Gabriel Falcão · Answer 9 · Tue May 25 2021 23:28:59 GMT+0800 (China Standard Time)

Thanks @ThiefMaster, I already merged 0eb4b16 and released the changes in version 1.1.3 if you could kindly test it :)

Gabriel Falcão · Answer 10 · Tue May 25 2021 23:29:38 GMT+0800 (China Standard Time)

@Gidgidonihah if you could also test your code against 1.1.3 that would be great.
Thanks folks!

Jason Weir · Answer 11 · Wed May 26 2021 03:43:02 GMT+0800 (China Standard Time)

@gabrielfalcao Same result for me. Thanks for getting on this quickly. I'll try to get an MWE together at the end of my sprint.

Jason Weir · Answer 12 · Fri May 28 2021 02:12:56 GMT+0800 (China Standard Time)

@gabrielfalcao Here's a gist for a MWE.

Using a python 3.9.5 virtualenv, just ran:

pip install -r requirements.txt
python -m unittest

Tim Schilling · Answer 13 · Fri May 28 2021 03:02:09 GMT+0800 (China Standard Time)

@gabrielfalcao It looks like there's another thing duplicating requests (I think). I've created a unit test within the repo that fails.

I narrowed it down to a POST that has data. Unless it's a preflight request that's being stored? That would explain the body being 0, though I'd expect the method to be OPTIONS and not POST.

Dave Randall · Answer 14 · Tue Jun 22 2021 17:51:32 GMT+0800 (China Standard Time)

@gabrielfalcao I can confirm that it only appears to impact POST requests as @tim-schilling suggested:

import httpretty
import requests

httpretty.enable()

httpretty.register_uri(method=httpretty.POST, uri="https://example.com/foo")
requests.post("https://example.com/foo", data="bar")
len(httpretty.latest_requests())
# returns 2

httpretty.reset()

httpretty.register_uri(method=httpretty.GET, uri="https://example.com/foo")
requests.get("https://example.com/foo")
len(httpretty.latest_requests())
# returns 1

Gabriel Falcão · Answer 15 · Tue Jun 22 2021 20:39:46 GMT+0800 (China Standard Time)

@gabrielfalcao I can confirm that it only appears to impact POST requests as @tim-schilling suggested:

import httpretty
import requests

httpretty.enable()

httpretty.register_uri(method=httpretty.POST, uri="https://example.com/foo")
requests.post("https://example.com/foo", data="bar")
len(httpretty.latest_requests())
# returns 2

httpretty.reset()

httpretty.register_uri(method=httpretty.GET, uri="https://example.com/foo")
requests.get("https://example.com/foo")
len(httpretty.latest_requests())
# returns 1

Thanks @daveygit2050 I'll incorporate this snippet into a new test case to reproduce the issue as soon as I can manage. Unfortunately I might not be able to attend to this until next week but I'll keep yall posted.

Adrian · Answer 16 · Wed Oct 27 2021 20:32:46 GMT+0800 (China Standard Time)

A release containing this fix would be great!

Andrew Ashbacher · Answer 17 · Wed Nov 03 2021 00:18:12 GMT+0800 (China Standard Time)

Thank you for creating a tool that helps improve automated tests! 🤗

I recently encountered this issue and wanted to add more info. In my testing, this applies to any requests (POST, PUT, DELETE, and/or PATCH) when they contain a request body. Below is a test file that demonstrates this behavior. (I did not include tests with request bodies for HTTP methods which do not allow request bodies (GET, HEAD, CONNECT, and OPTIONS), however these also demonstrate the behavior when a request body is provided.)

import httpretty
import urllib.request as request


class TestHTTpretty:
    @httpretty.activate(verbose=True, allow_net_connect=False)
    def run(self, method, data=None):
        url = 'https://example.com/foo'
        httpretty.register_uri(method=method, uri=url)
        req = request.Request(url=url, method=method, data=data)
        request.urlopen(req)
        assert len(httpretty.latest_requests()) == 1

    # the first eight tests with empty request bodies succeed
    def test_get(self):
        self.run(httpretty.GET)

    def test_head(self):
        self.run(httpretty.HEAD)

    def test_connect(self):
        self.run(httpretty.CONNECT)

    def test_options(self):
        self.run(httpretty.OPTIONS)

    def test_post_empty(self):
        self.run(httpretty.POST)

    def test_put_empty(self):
        self.run(httpretty.PUT)

    def test_delete_empty(self):
        self.run(httpretty.DELETE)

    def test_patch_empty(self):
        self.run(httpretty.PATCH)

    # the following four tests with populated request bodies fail
    def test_post_with_data(self):
        self.run(httpretty.POST, b'lorem ipsum')

    def test_put_with_data(self):
        self.run(httpretty.PUT, b'lorem ipsum')

    def test_delete_with_data(self):
        self.run(httpretty.DELETE, b'lorem ipsum')

    def test_patch_with_data(self):
        self.run(httpretty.PATCH, b'lorem ipsum')

Output:

============================ short test summary info ============================
FAILED test/httpretty_test.py::TestHTTpretty::test_post_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_put_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_delete_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_patch_with_data - assert 2 == 1

Cansin Yildiz · Answer 18 · Thu Mar 03 2022 16:09:45 GMT+0800 (China Standard Time)

I am dealing with the exact same issue. Looks like there was a fix merged into the master. As @ThiefMaster was pointing out, @gabrielfalcao would it be possible to deploy a new release?

Christian Assing · Answer 19 · Tue Nov 29 2022 18:27:17 GMT+0800 (China Standard Time)

+1 for a 1.1.5 release

Adrian · Answer 20 · Wed Jul 05 2023 21:54:42 GMT+0800 (China Standard Time)

@gabrielfalcao Can we please get an 1.1.5 release containing this fix?

Gabriel Falcão · Answer 21 · Thu Jul 06 2023 09:59:24 GMT+0800 (China Standard Time)

Salutations all, my macbook got damaged and I'm quite a bit limited at present due to the time taken for the new parts to arrive. I plan on applying the patch and resolve the situation as soon as the universe collaborates. Please hang in there and I'll report back as soon as humanely possible. -- Kind regards, Gabriel

Gabriel Falcão · Answer 22 · Sat Jul 08 2023 05:38:17 GMT+0800 (China Standard Time)

Hi folks,
I presently find myself somewhat able to work on this issue. I'll do what I can to make a release. Thanks for your patience so far.

Christian Assing · Answer 23 · Thu Dec 07 2023 18:05:09 GMT+0800 (China Standard Time)

Hi @gabrielfalcao,

This is a friendly reminder; I hope you'll find some time to make a new release :)

Thank you

Stephan Fuhrmann · Answer 24 · Wed Jan 03 2024 17:24:20 GMT+0800 (China Standard Time)

Problem exists in httpretty==1.1.4
httpretty==1.0.5 works as a workaround

@gabrielfalcao thanks for your great effort, it is appreciated!