gunicorn doesn't work

[Dennys Hsieh]

I can generate correct Prometheus metrics with gunicorn when it's worker is only 1. (I check the gauge is correct)
But when I increase the worker to 2, the gauge is not correct anymore.

And I add this environment, the result is the metric page is empty now.
export PROMETHEUS_MULTIPROC_DIR=/home/ubuntu/ap/tmp

I also try to add the code, but it's still show empty. Any suggestion?

from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)

[Steve Hillier]

Do you get any errors or any output on the metrics page at all?

The user running the application will need permission to write to PROMETHEUS_MULTIPROC_DIR, and the dir must already exist, but in my testing if it's a permission error, I get errors when trying to visit the metric page not empty metrics.

Just to confirm- are you using export PROMETHEUS_MULTIPROC_DIR=/home/ubuntu/ap/tmp as described to set the env var when running Gunicorn, or another method?

[Dennys Hsieh]

I use an Ubuntu account to run my program, but there is no error in console.
(I also use loguru, there is no exception in the log file ether.)
And I use bash and use export command.
Or is there any debug mode?

The metrics page is complete empty, I use F12 in Chrome but see nothing.

[Steve Hillier]

Hmm ok, I haven't run into this before. Are you able to post the bash script you use to start gunicorn so I can try to reproduce the problem?

Also, can you confirm you are using the metrics endpoint handler from this library? e.g.:

from fastapi import FastAPI
from starlette_exporter import PrometheusMiddleware, handle_metrics

app = FastAPI()
app.add_middleware(PrometheusMiddleware)
app.add_route("/metrics", handle_metrics)

The reason I ask is that the metrics handler will handle multiprocessing, but if you make your own handler you might have to add in a couple lines.

[Dennys Hsieh]

Thanks for your support, the following is my Python code

from fastapi import FastAPI, File, UploadFile, status, Body, HTTPException
from fastapi.responses import FileResponse, HTMLResponse
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI(title='CropApp', version='2.0')
app.add_middleware(
    CORSMiddleware,
    allow_origin_regex='https?://.*',
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)
app.add_middleware(
     PrometheusMiddleware,
     app_name="CropApp",
     skip_paths=['/metrics'],
     buckets=[1, 2, 2.8, 2.9, 3, 3.1, 3.2, 3.3, 3.4, 5, 6, 7, 8, 9, 10])
app.add_route("/metrics", handle_metrics)

And this is my shell script. When I change -w 1 to -w 2, /metrics will be empty.

#!/bin/bash
export PROMETHEUS_MULTIPROC_DIR=/home/ubuntu/app/tmp
gunicorn app:app -w 1 -b 0.0.0.0:8080 -k uvicorn.workers.UvicornWorker --statsd-host=localhost:9125 --statsd-prefix=argi --log-level debug

[Steve Hillier]

Unfortunately the code works as expected for me. Even with -w 2 I see metrics as usual. The only changes I made were to use a directory that exists on my machine (i.e. my home dir not /home/ubuntu), and removing the --statsd-* args from the start script.

I am actually confused about why you are seeing a difference between -w 1 and -w 2, since simply having PROMETHEUS_MULTIPROC_DIR set should trigger the Prometheus multiprocess behavior (regardless of the number of workers).

[Steve Hillier]

Do you see the metrics files being written to /home/ubuntu/app/tmp? That dir should have files like counter_1405908.db, histogram_1407491.db, etc.

[Dennys Hsieh]

Hi, I think I know the problem, the following is my test cases.

Case 1:

1. Run App with -w 1
2. Connect to http://x.x.x.x/metrics => I see some metrics. (I don't run my app yet.)

Case 2:

1. Run App with -w 2
2. Connect to http://x.x.x.x/metrics => I see empty.
3. Run my app.
4. Connect to http://x.x.x.x/metrics again => I see some metrics and files like counter_1405908.db are generated.