Giters

stephenhillier / starlette_exporter

Prometheus exporter for Starlette and FastAPI

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Counter decreasing

owlas opened this issue · comments

commented

This might be a deployment issue but any info could help.

I'm using starlette_exporter with a fastapi app in k8s. Currently I have just a single pod running. The metrics that get scraped keep flicking between two values.

For example:

starlette_requests_total{method="GET", path="/hello/{hello_id}", status_code="200"} 19.0
starlette_requests_total{method="GET", path="/hello/{hello_id}", status_code="200"} 11.0
starlette_requests_total{method="GET", path="/hello/{hello_id}", status_code="200"} 11.0
starlette_requests_total{method="GET", path="/hello/{hello_id}", status_code="200"} 11.0
starlette_requests_total{method="GET", path="/hello/{hello_id}", status_code="200"} 19.0
starlette_requests_total{method="GET", path="/hello/{hello_id}", status_code="200"} 19.0
starlette_requests_total{method="GET", path="/hello/{hello_id}", status_code="200"} 11.0

Where would these multiple states be stored? Could this be a problem with multi threading in fastapi?

commented

Hi there, thanks for the report.

Are you using gunicorn? At first glance it sounds like you may need to try setting prometheus_multiproc_dir. https://github.com/prometheus/client_python#multiprocess-mode-gunicorn

Please let me know if that helps or if you've already tried that (if that does solve the issue I should put something in the readme).

commented

Hey @stephenhillier, Yes I'm using gunicorn and uvicorn. This looks really promising.

I'll take a look now and let you know if I get it working. I can open a PR on the docs if you think this is a common enough use case.

commented

Now that a section has been added to the docs I think this can be closed. Feedback welcome if anyone is still having an issue and /or the docs need more explanation.
https://github.com/stephenhillier/starlette_exporter#multiprocess-mode-gunicorn-deployments