Share a numpy array in gunicorn processes

Question

I have a big numpy array that is stored in redis. This array acts as an index. I want to serve filtered result over http from a flask app running on gunicorn and I want all the workers spawned by gunicorn to have access to this numpy array. I don't want to go to redis every time and deserialize the entire array in memory, instead on startup I want to run some code that does this and every forked worker of gunicorn just gets a copy of this array. The problem is, I can not find any examples on how to use gunicorn's server hooks: http://docs.gunicorn.org/en/latest/configure.html#server-hooks to achieve this. May be server hooks is not the right way of doing it, has anyone else done something similar?

Answer 1

Create an instance of a Listener server and have your gunicorn children connect to that process to fetch whatever data they need as Clients. This way the processes can modify the information as needed and request it from the main process instead of going to Redis to reload the entire dataset.

More info here: Multiprocessing - 16.6.2.10. Listeners and Clients.