a2wsgi can't work with BaseHTTPMiddleware in Starlette
fnep opened this issue · comments
For the sake of backwards compatibility, I'm running a complicated stack of apache (mod_wsgi) -> a2wsgi (ASGIMiddleware) -> fastapi, and with the update of fastapi==0.107.0 this broke, after they have internally switched to a relatively new version of Starlette (0.28.0).
In apache I'm getting the error RuntimeError: Unexpected message received: http.request, but this is here only to make this issue easier to find later.
I now tried to reduce this stack to make it easier to understand and think the a2wsgi part might be the issue or at least you may know what it actually is.
Based on the starlette hello-world i added a custom middleware and the a2wsgi.ASGIMiddleware. Removing the custom middle middleware fixes the issue (but removes the functionality).
from starlette.applications import Starlette
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.middleware import Middleware
from starlette.responses import JSONResponse
from starlette.routing import Route
from a2wsgi import ASGIMiddleware
class HeaderAddTestMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
response = await call_next(request)
response.headers["X-Test"] = "Test"
return response
async def homepage(request):
return JSONResponse({"hello": "world"})
app = Starlette(
debug=True,
middleware=[
Middleware(HeaderAddTestMiddleware),
],
routes=[
Route("/", homepage),
],
)
application = ASGIMiddleware(app)To run it i use uwsgi like this uwsgi --http :8000 --wsgi-file main.py.
Using starlette==0.27.0 and a2wsgi==1.10.0 this just works, but with starlette==0.28.0 and later (current version is starlette==0.37.0) i get this error on request.
❯ curl -v http://127.0.0.1:8000
* Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> GET / HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-length: 17
< content-type: application/json
< x-test: Test
<
* Connection #0 to host 127.0.0.1 left intact
{"hello":"world"}
[...]
[pid: 17764|app: 0|req: 28/28] 127.0.0.1 () {28 vars in 291 bytes} [Wed Feb 7 09:22:15 2024] GET / => generated 17 bytes in 5 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
Traceback (most recent call last):
File ".venv/lib/python3.12/site-packages/a2wsgi/asgi.py", line 235, in __call__
yield from self.error_response(start_response, message["exception"])
File ".venv/lib/python3.12/site-packages/a2wsgi/asgi.py", line 266, in error_response
start_response(
OSError: headers already sent
[pid: 17764|app: 0|req: 29/29] 127.0.0.1 () {28 vars in 291 bytes} [Wed Feb 7 09:22:17 2024] GET / => generated 17 bytes in 7 msecs (HTTP/1.1 200) 2 headers in 71 bytes (2 switches on core 0)
[...]
The hello-world response gets shipped anyway, and even the header is set as wanted. It also gives this error when i just return await call_next(request) in the dispatch method without actually modifying any header.
I would normally assume the issue is somewhere in BaseHTTPMiddleware, but running the same code using asgi via uvicorn (uvicorn main:app) it also works.
From the error, it seems that this is caused by an exception in your Starlette. You can use a DebugMiddleware to debug your application. This is not a problem with a2wsgi; it complies with the WSGI standard here.
Ok, thank you for your assessment. So far, all my debugging attempts failed.
I will probably raise the same question at Starlette then, because the code above is almost directly copied from their docs (example app and custom middleware).
To be honest, I expect them to also reject it because running the asgi-app works, and only running the a2wsgi-wrapped-wsgi app fails, and so they will probably not see it as their business, too. As a basic user of those tools, with only limited knowledge about how it is working, it is relatively hard to argue in any direction here. :)
Please do not close. China is celebrating the Chinese New Year, and I will come to debug this issue after my Spring Festival holiday ends. (By the way, I am also one of the maintainers of Starlette.)
Oh, sorry, I thought you left it open in case I had something else to add. Of course.
I was not aware you are also working on Starlette. Thank you for all the time and effort.
We are running currently into the same issue when trying to update FastAPI to 0.109.2 and already investigated the issue that we figured out that currently a2wsgi is not compatible to FastAPI of the latest version if you are using BaseHTTPMiddleware (from Starlette). I did not go into detail yet where the underlying issue is, but happy to follow up & provide some more insights if its helpful ✌️
@LysanderKie Dependent on what you do and how urgent it is, you might switch to using an pure asgi middleware like this in the meantime: https://www.starlette.io/middleware/?cmdf=starlette+async+middleware#inspecting-or-modifying-the-response
{'type': 'http.request', 'body': b'', 'more_body': False}
Traceback (most recent call last):
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\middleware\errors.py", line 186, in __call__
raise exc
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\middleware\errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\middleware\base.py", line 192, in __call__
await response(scope, wrapped_receive, send)
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\responses.py", line 265, in __call__
await wrap(partial(self.listen_for_disconnect, receive))
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\responses.py", line 261, in wrap
await func()
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\responses.py", line 238, in listen_for_disconnect
message = await receive()
File "C:\Users\aber\Documents\GitHub\a2wsgi\__pypackages__\3.10\lib\starlette\middleware\base.py", line 56, in wrapped_receive
raise RuntimeError(f"Unexpected message received: {msg['type']}")
RuntimeError: Unexpected message received: http.requestIt seems Starlette's BaseHTTPMiddleware requires some specific events.
Please test version 1.10.1. This should fix the problem.
Unfortunately, the code above with a2wsgi==1.10.1 and starlette==0.37.1, started with uwsgi as described, is just hanging in the curl request for me.
❯ curl -v http://127.0.0.1:8000
* Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> GET / HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.4.0
> Accept: */*
>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server
Unfortunately, I will continue to investigate this issue.
I found the problem. It seems to be caused by the blocking of "receive". EOF is not passed to the WSGI application through wsgi.input. I'm thinking about how to handle this strange issue.
After careful investigation, I finally confirmed that the problem lies in the handling of disconnect by BaseHTTPMiddleware. Because in the synchronous model of WSGI, the client never sends EOF, so the entire application will be stuck here. If necessary, I will make modifications in Starlette. This issue will not be closed until the problem is thoroughly resolved.
1.10.2 no longer hangs, but still cannot be used with BaseHTTPMiddleware. If you are very eager to complete the code, you can use Pure-ASGI middleware instead.
@abersheeran Do you've an Idea on how to solve this annoying issue? I also use a2wsgi with Starlette, since Starlette 0.27 we can no longer update the versions of these libraries. I'm trying to understand how everything works, but at this stage, the complexity of the different layers is beyond my comprehension.
@abersheeran Do you've an Idea on how to solve this annoying issue? I also use a2wsgi with Starlette, since Starlette 0.27 we can no longer update the versions of these libraries. I'm trying to understand how everything works, but at this stage, the complexity of the different layers is beyond my comprehension.
The current recommendation is not to use BaseHTTPMiddleware. As far as I know, Starlette also plans to deprecate BaseHTTPMiddleware. encode/starlette#2160
Thank you for the quick reply @abersheeran. After digging into my code I found a custom middleware based on the BaseHTTPMiddleware class. By removing this one from the list of middlewares everything is working fine now.