stephenhillier / starlette_exporter

Looks like PrometheusMiddleware might misbhehave when Starlette AuthenticationMiddleware raises an exception -- for example because a database or some other required resource is down.

It looks like finally block has a return statement, and the effect is that:

other middleware won't see the exception
Starlette default error handler does not see the error
ASGI server (e.g. uvicorn needs to catch the invalid behaior ("ASGI callable returned without starting response")

This happens because there is a return statement in finally block:

starlette_exporter/starlette_exporter/middleware.py

Lines 332 to 343 in 7b011cd

    
           finally: 
        
               # Decrement 'requests_in_progress' gauge after response sent 
        
               self.requests_in_progress.labels( 
        
                   method, self.app_name, *default_labels 
        
               ).dec() 
        
               if self.filter_unhandled_paths or self.group_paths: 
        
                   grouped_path = self._get_router_path(scope) 
        
                   # filter_unhandled_paths removes any requests without mapped endpoint from the metrics. 
        
                   if self.filter_unhandled_paths and grouped_path is None: 
        
                       return

POC, more or less the same style as Starlette docs + unit tests:

from starlette.applications import Starlette
from starlette.authentication import (
    AuthCredentials,
    AuthenticationBackend,
    AuthenticationError,
    SimpleUser,
    UnauthenticatedUser,
    requires,
)
from starlette.middleware import Middleware
from starlette.middleware.authentication import AuthenticationMiddleware
from starlette.responses import JSONResponse
from starlette.routing import Route
from starlette_exporter import PrometheusMiddleware


class PocAuthBackend(AuthenticationBackend):
    async def authenticate(self, request):
        if "Authorization" not in request.headers:
            return None

        auth_scheme, _, auth_token = request.headers["Authorization"].partition(" ")
        if auth_scheme != "token":
            raise AuthenticationError("Invalid authorization")

        scopes: list[str] = []
        if auth_token == "beef":
            user = SimpleUser(username="bobby")
            scopes = ["authenticated"]
        elif "raise" in auth_token:
            # Pretend that actual token check failed (e.g. DB connection error)
            raise ValueError("Failed")
        else:
            user = UnauthenticatedUser()
            scopes = []

        return AuthCredentials(scopes), user


@requires("authenticated")
async def hello(request):
    return JSONResponse(
        {
            "authenticated": request.user.is_authenticated,
            "user": request.user.display_name,
        },
    )


app = Starlette(
    routes=[
        Route("/hello", hello),
    ],
    middleware=[
        Middleware(
            PrometheusMiddleware,
            app_name="poc",
            prefix="poc",
            group_paths=True,
            filter_unhandled_paths=True,
        ),
        Middleware(
            AuthenticationMiddleware,
            backend=PocAuthBackend(),
        ),
    ],
)

Running the server:

$ uvicorn expoc:app

Sequence of requests:

$ curl localhost:8000/hello
Forbidden

$ curl -H "Authorization: token beef" localhost:8000/hello
{"authenticated":true,"user":"bobby"}

$ curl -H "Authorization: token dead" localhost:8000/hello       
Forbidden

$ curl -H "Authorization: token raise" localhost:8000/hello
Internal Server Error

Server logs:

INFO:     127.0.0.1:48628 - "GET /hello HTTP/1.1" 403 Forbidden
INFO:     127.0.0.1:55530 - "GET /hello HTTP/1.1" 200 OK
INFO:     127.0.0.1:42388 - "GET /hello HTTP/1.1" 403 Forbidden
ERROR:    ASGI callable returned without starting response.
INFO:     127.0.0.1:42392 - "GET /hello HTTP/1.1" 500 Internal Server Error

The error logging comes from uvicorn, meaning that Starlette error handling did not see the exception. Also any other middleware like Sentry would not be able to see it.

@jvtm Thank you for reporting this.

ERROR: ASGI callable returned without starting response. - I do want to address this error and fix the return (as you noted the return is interfering with the exception occurring), but I also think this is a secondary side effect of a bigger problem. The code path with the return statement is only hit because something else has already gone wrong (specifically, the request scope is missing information about the application routes when AuthenticationMiddleware raises the ValueError).

Until I can find a solution, can you handle these ValueErrors (or whatever they are) and convert them to AuthenticationErrors, raise HTTPExceptions or return a response like PlainTextResponse(str(exc), status_code=500)?

I have a workaround in place in the actual code, but it 's a bit clumsy because I need to generate also artificial events for Sentry and for metrics reporting (that would otherwise work out of the box, both being ASGI middleware or something similar).

So, not an urgent issue now that we know about it. Finally had some time for writing the minimal code reproducing the issue. :)

Thanks for checking.

For anyone else reading this, these might be relevant:

There's also a few authentication plugins (for checking how they possibly behave in case of errors) listed in:

https://www.starlette.io/third-party-packages/

Here is a fix: #81

	finally:
	# Decrement 'requests_in_progress' gauge after response sent
	self.requests_in_progress.labels(
	method, self.app_name, *default_labels
	).dec()

	if self.filter_unhandled_paths or self.group_paths:
	grouped_path = self._get_router_path(scope)

	# filter_unhandled_paths removes any requests without mapped endpoint from the metrics.
	if self.filter_unhandled_paths and grouped_path is None:
	return

Starlette AuthenticationBackend exceptions disappear