Question and ideas regarding multiprocess mode

[Michał Jaworski]

Hi, I have customer that runs sanic on production and convinced them to run their own prometheus stack. Unfortunately they run sanic in mutliprocessing mode and and that cannot be simply changed in their current architecture. I did try using vanilla prometheus-sanic with the settings recommended in the README with monitor.expose_endpoint() but unfortunately could get working result. Metrics endpoint simply returned empty endpoint:

$http :80/metrics
HTTP/1.1 200 OK
Connection: close
Content-Length: 0
Content-Type: text/plain; version=0.0.4; charset=utf-8

Did anyone had success in running prometheus-sanic in multiprocess mode?

Also requirement to run metrics endpoint on the same port as the service itself kinda sucks because rest of their services is integrated with dedicated metrics port. What I did instead is to prebind metrics port and make sanic to start the build in HTTP deamon thread after actual worker start. Following is excerpt from the code we use currently:

from prometheus_client.registry import REGISTRY
from prometheus_client.exposition import _ThreadingSimpleServer, MetricsHandler

def prebind_metrics_server(port, addr='', registry=REGISTRY):
    custom_metrics_handler = MetricsHandler.factory(registry)
    httpd = _ThreadingSimpleServer((addr, port), custom_metrics_handler)
    httpd_thread = threading.Thread(target=httpd.serve_forever)
    httpd_thread.daemon = True
    return httpd_thread

# later in app init
metrics_server = prebind_metrics_server(8080)

@app.listener('after_server_start')
async def _(*_):
    # pre-bound metrics server must be started after
    # actual sanic worker has started
    metrics_server.start()

app.run()

The only thing that I'm missing to make it fully functional is to be able to preconfigure the registry object used in monitor.

[Denis Malin]

Yes, I'm initializing the folder for metrics

python:

import os
os.environ['prometheus_multiproc_dir'] = 'prometheus_tmp'

or (sh)

export prometheus_multiproc_dir=prometheus_tmp

you also need to create a folder, it needs to be cleaned when the app starts

---

example:

https://github.com/skar404/prometheus-sanic/blob/master/examples/multiprocess_base_app.py