Using trio with httpx in a basic example works. In a basic locust script it fails with `NotImplementedError: unsupported platform`
dkelsey opened this issue · comments
Description:
- I have a basic example (
testTrio.py
) that useshttpx.AsyncClient
withtrio
. This works correctly when I run it. - I have a basic
locustfile.py
that also useshttpx.AsyncClient
. It fails withNotImplementedError: unsupported platform
.
Working Script testTrio.py
import httpx
import trio
url = 'https://puginarug.com/'
async def main():
async with httpx.AsyncClient() as client:
response = await client.get(url)
print(response)
trio.run(main)
Output
<Response [200 OK]>
Non-working locustfile.py
from locust import HttpUser, task
import trio, httpx
class MyUser(HttpUser):
wait_time = between(1, 3)
url = 'https://puginarug.com/'
async def on_start(self):
# Create an httpx.AsyncClient session at the start of the user's lifecycle
self.client = httpx.AsyncClient(backend='asyncio')
async def on_stop(self):
# Close the httpx.AsyncClient session at the end of the user's lifecycle
await self.client.aclose()
@task
async def my_task(self):
# Use httpx.AsyncClient to make asynchronous requests
response = await self.client.get(url)
print(f"Response: {response.text}")
Output
Traceback (most recent call last):
File "/Users/dkelsey/.pyenv/versions/HTTPX/bin/locust", line 8, in <module>
sys.exit(main())
^^^^^^
File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/locust/main.py", line 100, in main
docstring, _user_classes, shape_classes = load_locustfile(_locustfile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/locust/util/load_locustfile.py", line 70, in load_locustfile
loader.exec_module(imported)
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/Users/dkelsey/Development/DES/load-testing-tools/experiments/AsyncHTTXClient/locustfile.2.py", line 2, in <module>
import trio, httpx
File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/__init__.py", line 22, in <module>
from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED # isort: split
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/_core/__init__.py", line 21, in <module>
from ._local import RunVar, RunVarToken
File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/_core/_local.py", line 9, in <module>
from . import _run
File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/_core/_run.py", line 2800, in <module>
raise NotImplementedError("unsupported platform")
NotImplementedError: unsupported platform
Environment:
OS: Mac M1
Python: 3.11.4 (venv)
Locust:
Name: locust
Version: 2.23.1
Summary: Developer friendly load testing framework
Home-page:
Author:
Author-email:
License: MIT
Location: /Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages
Requires: ConfigArgParse, flask, Flask-Cors, Flask-Login, gevent, geventhttpclient, msgpack, psutil, pyzmq, requests, roundrobin, Werkzeug
Required-by:
httpx:
Name: httpx
Version: 0.27.0
Summary: The next generation HTTP client.
Home-page:
Author:
Author-email: Tom Christie <tom@tomchristie.com>
License:
Location: /Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages
Requires: anyio, certifi, httpcore, idna, sniffio
Required-by:
trio:
Name: trio
Version: 0.24.0
Summary: A friendly Python library for async concurrency and I/O
Home-page:
Author:
Author-email: "Nathaniel J. Smith" <njs@pobox.com>
License: MIT OR Apache-2.0
Location: /Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages
Requires: attrs, idna, outcome, sniffio, sortedcontainers
Required-by:
Line 2800
of trio/_core/_run.py
![Screenshot 2024-02-28 at 9 57 13 PM](https://private-user-images.githubusercontent.com/3906047/308811413-0ee2086f-6137-40e2-98db-4b8cee029ae6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI3NzkyMTIsIm5iZiI6MTcxMjc3ODkxMiwicGF0aCI6Ii8zOTA2MDQ3LzMwODgxMTQxMy0wZWUyMDg2Zi02MTM3LTQwZTItOThkYi00YjhjZWUwMjlhZTYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDQxMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA0MTBUMTk1NTEyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Y2RhNTUxMmE1M2FlZGYwYjA2ZTkzZjk0OTA0MzA0OGM3NjFjNDQxNjFjNTBmMGU2Nzc4NmU2MjkxNTViOWQ1YSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.jzlwZcIJ563m1JpOhExglemnh7lt21pzopOkG84sSx8)
Questions
- Why would the basic example work and locust not?
- Is this related to #2960
I've had success using httpx with more complicated trio applications, so I'm thinking locust must do something very strange. Probably not the same thing as gunicorn though (re your question: gunicorn fails because it removes the primitives we need to wait for files).
EDIT: nevermind, locust does use gevent. see https://github.com/locustio/locust/blob/baf007f1c9f243a550621ecc99720d835513e220/docs/changelog.rst#L789-L790
There's no satisfying solution IMO because if trio reaches into gevent's API to pull out the original kqueue, then I think we're blocking the greenlet? Which is obviously not what the user wants IMO.
Maybe we should just fall back to gevent's patched version of something? I'm not too familiar with the underlying mechanics of trio to say what. Hopefully someone else has an answer!
EDIT 2: Heh, looks like they did some special casing on their end re: gevent w/ trio! See locustio/locust@aedc3b9
sounds like something to bring up on their end in that case. Either to re-fix their workaround, or to see if they know how we should work around the patch on our end.
Is this a thing that's broken in one of our recent releases, given that we're getting several reports now? Or new version of gevent/other?
Is it possible to do backend='trio'
?
@CoolCat467 is this something you are asking me to try?
I added backent'trio'
as a param to httpx.AsyncClient
I get the same errror.
I ran both scripts in debug (pyCharm).
TestTrio.py
It lands on this part of trio/_core/_run.py
...
elif TYPE_CHECKING or hasattr(select, "kqueue"):
from ._generated_io_kqueue import *
from ._io_kqueue import (
EventResult as EventResult,
KqueueIOManager as TheIOManager,
_KqueueStatistics as IOStatistics,
)
TYPE_CHECKING
isFalse
- The call to
hasattr
inspectsselect-cpython-311-darwin.so
and findskqueue
exists.
<module 'select' from '/Users/dkelsey/.pyenv/versions/3.11.4/lib/python3.11/lib-dynload/select.cpython-311-darwin.so'>
![Screenshot 2024-03-05 at 10 35 40 PM](https://private-user-images.githubusercontent.com/3906047/310386963-b5b564a3-60fe-43b5-849a-11be50050b63.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI3NzkyMTIsIm5iZiI6MTcxMjc3ODkxMiwicGF0aCI6Ii8zOTA2MDQ3LzMxMDM4Njk2My1iNWI1NjRhMy02MGZlLTQzYjUtODQ5YS0xMWJlNTAwNTBiNjMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDQxMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA0MTBUMTk1NTEyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MzE2NGFkZjZhMzAzYzUxNjBhMmNlNmE1ODBlNzFhYzYwN2NjOWEyZjgwN2IzZDUzOThjZDZkYTQ2ZDM1Njg4YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.OArS7wB0nEiy9S1kT5Nr1SxEdJGk7JiI_ttVeZsgZWA)
- it exists.
- it works.
locustfile.py
I place a break point in _run.py
and ran the locust script in debug.
The same check occurs:
<module 'select' from '/Users/dkelsey/.pyenv/versions/3.11.4/lib/python3.11/lib-dynload/select.cpython-311-darwin.so'>
![Screenshot 2024-03-05 at 10 31 09 PM](https://private-user-images.githubusercontent.com/3906047/310385910-27452a98-3415-4ded-b72a-ed89ce7e26ca.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI3NzkyMTIsIm5iZiI6MTcxMjc3ODkxMiwicGF0aCI6Ii8zOTA2MDQ3LzMxMDM4NTkxMC0yNzQ1MmE5OC0zNDE1LTRkZWQtYjcyYS1lZDg5Y2U3ZTI2Y2EucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDQxMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA0MTBUMTk1NTEyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YjgyMGRkYmI5YjY3NjhjMDQ5MjJlZDc5ZjQwNzViZWQ4M2FmNzIzNzVhOTE4MjU5NTExM2ZhMjg3ZjVkMDc1YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.vCrclMZCC-4q_qdt0okfG9gfrLwNIz-0kwIWtHCpwOE)
- no
kqueue
- It fails.
So where did kqueue
go?
It was monkey patched away?
Note:
I followed this to get debugging working with locust/gevent in pyCharm:
Also added this to the locust script at the end (and imported it):
if __name__ == "__main__":
run_single_user(MyUser)
After all this... I re-read the comments from @A5rocks and @jakkdl
I now understand what you wrote.
So how to get httpx.AsyncClient
or trio
to work in locust
?
A hacky solution would be to edit /Users/dkelsey/.pyenv/versions/HTTPX/bin/locust
to try import trio
before anything else, which will cache trio
. Obviously this is a no-go for various reasons.
The best solution here (IMO) is to use non-async httpx, and use gevent if you need "async" stuff. Unfortunately it looks like locust is built around this.
As for fixing this in trio, I'm starting to lean more towards the broken-solution to this where we reach into gevent and pull out kqueue
(there's a way to do that) but this probably requires more thought.
Actually I thought about this a bit more, and select.kqueue = gevent.monkey.get_original("select", "kqueue")
before you import trio (with the necessary imports, of course) might work in your locustfile.py
? I'm on mobile so I can't check whether you can assign to module contents like that.
Obviously this is a bad solution but less so than my previous one where you had to patch locust!
Found this in the locust docs here: testing other systems
It is important that the protocol libraries you use can be monkey-patched by gevent.
A hacky solution would be to edit
/Users/dkelsey/.pyenv/versions/HTTPX/bin/locust
to tryimport trio
before anything else, which will cachetrio
. Obviously this is a no-go for various reasons.The best solution here (IMO) is to use non-async httpx, and use gevent if you need "async" stuff. Unfortunately it looks like locust is built around this.
As for fixing this in trio, I'm starting to lean more towards the broken-solution to this where we reach into gevent and pull out
kqueue
(there's a way to do that) but this probably requires more thought.
My intention is to send multiple multiplexed requests for resources through a single connection.
I'll reiterate that I think you should open an issue in locust and see if they can re-fix their workaround. Getting kqueue
support into gevent
or general gevent
support in trio
both sound like quite tough tasks.
Thanks @jakkdl those are good recommendations.
I found a way to achieve what I want useing pycurl in locust, writing a client and user class following the locust docs on testingnother services.
I'm satisfied. Thanks again.