Giters

getsentry / responses

A utility for mocking out the Python Requests library.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

responses not mocking urls for concurrent.futures.ProcessPool

GeeCastro opened this issue · comments

commented

Describe the bug

Reponses works with ThreadPoolExecutor but doesn't with ProcessPoolExecutor. I'm guessing it may be because the underlying process doesn't get the responses setup when spawned? But just in case I thought I'd ask if there was a solution for my tests.

Additional context

No response

Version of responses

0.23.3

Steps to Reproduce

from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import ThreadPoolExecutor
import requests
import responses

@responses.activate
def test_get_threading():
    url = "http://test.org/get"
    responses.get(url, json={"status": "ok"})

    with ThreadPoolExecutor() as pool:
        r = pool.map(requests.get, [url, url])

    results = list(r)
    assert len(results) == 2
    assert results[0].json() == {"status": "ok"}
    assert results[1].json() == {"status": "ok"}

@responses.activate
def test_get_multiprocessing():
    url = "http://fake.org/get"
    responses.get(url, json={"status": "ok"})

    with ProcessPoolExecutor() as pool:
        r = pool.map(requests.get, [url, url])    # tries to reach non existant `http://test.org/get`

    results = list(r)
    assert len(results) == 2
    assert results[0].json() == {"status": "ok"}
    assert results[1].json() == {"status": "ok"}


if __name__ == "__main__":
    print("Running threading...")
    test_get_threading()
    print("Running multiprocessing...")
    test_get_multiprocessing()

runs with pytest or main and throws an error:

Details 
python3.10 test_concurrent.py
Running threading...
Running multiprocessing...
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 415, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1046ec5b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='test.org', port=80): Max retries exceeded with url: /get (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1046ec5b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/process.py", line 205, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/process.py", line 205, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='test.org', port=80): Max retries exceeded with url: /get (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1046ec5b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/tests/test_concurrent.py", line 35, in <module>
    test_get_multiprocessing()
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/.venv/lib/python3.10/site-packages/responses/__init__.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/Users/gauthiercastro/code/ml-sc-training-pipeline/tests/test_concurrent.py", line 14, in test_get_multiprocessing
    results = list(r)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/process.py", line 575, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/Users/gauthiercastro/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
requests.exceptions.ConnectionError: None: Max retries exceeded with url: /get (Caused by None)

Expected Result

Mocking urls for both ThreadPoolExecutor and ProcessPoolExecutor

Actual Result

Mocking urls for both ThreadPoolExecutor but reaches actual website with ProcessPoolExecutor

EDIT: make 2d test fail

commented

hmm, cannot answer immediately, need to perform research.

but indeed that shows that the request was passed through

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='test.org', port=80): Max retries exceeded with url: /get (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1046ec5b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
commented

@Chichilele I did run an example and it works fine for me:

issue3.py::test_get_threading PASSED                                     [ 50%]
issue3.py::test_get_multiprocessing PASSED                               [100%]

although, I had to modify the line to:

    with ProcessPoolExecutor() as pool:
        r = pool.map(requests.get, [url, url])
commented

no reply, closing as cannot reproduce

commented

Apologies I missed the notification. Can you check that this repo fails when running pytest? https://github.com/Chichilele/reponses-thread-issue

The original post was missing calling the response

commented

Please open a bug filling the form