Collecting prometheus metrics from a separate port using flask and gunicorn with multiple workers

Question

I run a small flask application with gunicorn and multiple worker processes on kubernetes. I would like to collect metrics from this application with prometheus, but the metrics should only be accessible cluster internally on a separate port (as this required in our current setting).

For one gunicorn worker process I could use the start_http_server fucntion from the python client library to expose metrics on a different port than the flask app.

A minimal example may look like this:

from flask import Flask
from prometheus_client import start_http_server, Counter

NUM_REQUESTS = Counter("num_requests", "Example counter")

app = Flask(__name__)

@app.route('/')
def hello_world():
    NUM_REQUESTS.inc()
    return 'Hello, World!'

start_http_server(9001)

To start the app do the following:

gunicorn --bind 127.0.0.1:8082 -w 1 app:app

However this only works for one worker process.

In the documentation of the client library is also a section on how to use prometheus and gunicorn with multiple worker processes by specifying a shared directory for the worker processes as an environment variable where metrics are written to (prometheus_multiproc_dir).

So following the documentation the above example for multiple workers would be:

A gunicorn config file:

from prometheus_client import multiprocess

def worker_exit(server, worker):    
    multiprocess.mark_process_dead(worker.pid)

The application file:

import os
from flask import Flask
from prometheus_client import Counter

NUM_REQUESTS = Counter("num_requests", "Example counter")

app = Flask(__name__)

@app.route('/')
def hello_world():
    NUM_REQUESTS.inc()
    return "[PID {}]: Hello World".format(os.getpid())

To start the app do:

rm -rf flask-metrics/
mkdir flask-metrics
export prometheus_multiproc_dir=flask-metrics
gunicorn --bind 127.0.0.1:8082 -c gunicorn_conf.py -w 3 app:app

However in this setting I don't really know how to accesses the metrics stored in flask-metrics on a separate port. Is there way to get this done?

I am a bit new to these things, so if I am approaching the problem in the wrong way I am also happy for advice what would be the best way to address my case.

Answer 1

What you would want to do here is start up a separate process just to serve the metrics. Put the app function in https://github.com/prometheus/client_python#multiprocess-mode-gunicorn in an app of its own, and make sure that prometheus_multiproc_dir is the same for both it and the main application.