Menu
I'm running a Flask application that is basically pulling tweets from Twitter. While running the app with the embedded Flask server gives no troubles, when running within gUnicorn I get duplicated tweets, mostly because I have 2 threads receiving the callback from Twitter.
For instance, if I start my app using
python app.py
When receiving tweets I'm getting this expected output, see that I've attached thread info (first param) in the logger output:
140721974449920 2015-03-12 17:59:13,030 INFO: Got message from streaming Twitter API! [in /home/mosquito/git/opencoast_streamer/app.py:83]
140721974449920 2015-03-12 17:59:14,646 INFO: Got message from streaming Twitter API! [in /home/mosquito/git/opencoast_streamer/app.py:83]
140721974449920 2015-03-12 17:59:49,031 INFO: Got message from streaming Twitter API! [in /home/mosquito/git/opencoast_streamer/app.py:83]
As you can see, timestamp looks valid too, checking at the mongo collection where I'm storing this, I see documents are OK. Then, if I start the app using gunicorn:
gunicorn app:app -b localhost:8000 --debug
And then check the logs, I can see that 2 different threads are getting data:
139883969844992 2015-03-12 17:52:05,104 INFO: Got message from streaming Twitter API! [in /home/mosquito/git/opencoast_streamer/app.py:83]
139883961452288 2015-03-12 17:52:05,106 INFO: Got message from streaming Twitter API! [in /home/mosquito/git/opencoast_streamer/app.py:83]
139883969844992 2015-03-12 17:53:36,480 INFO: Got message from streaming Twitter API! [in /home/mosquito/git/opencoast_streamer/app.py:83]
139883961452288 2015-03-12 17:53:36,481 INFO: Got message from streaming Twitter API! [in /home/mosquito/git/opencoast_streamer/app.py:83]
As you can see something weird is going on....then I went to see and check gunicorn:
ps aux | grep gunicorn
mosquito 25035 3.1 0.3 54612 12516 pts/1 S 15:31 0:01 /home/mosquito/www/env/bin/python /home/mosquito/www/env/bin/gunicorn app:app -b localhost:8000
mosquito 25606 0.0 0.4 66904 17016 pts/1 R 15:32 0:00 /home/mosquito/www/env/bin/python /home/mosquito/www/env/bin/gunicorn app:app -b localhost:8000
mosquito 25610 0.0 0.0 13220 956 pts/3 S+ 15:32 0:00 grep --color=auto gunicorn
Thus, I'm starting to think that this has to do with gUnicorn...any ideas why gUnicorn is spawining 2 process for my Flask app?
Thanks!
957
If each request takes 10 milliseconds, a single worker dishes out 100 RPS. If some requests take 10 milliseconds, others take, say, up to 5 seconds, then you'll need more than one concurrent worker, so the one request that takes 5 seconds does not "hog" all of your serving capability.
Gunicorn is based on the pre-fork worker model. This means that there is a central master process that manages a set of worker processes. The master never knows anything about individual clients. All requests and responses are handled completely by worker processes.
Flask will process one request per thread at the same time. If you have 2 processes with 4 threads each, that's 8 concurrent requests.
I believe this is not gUnicorn's fault but rather the intended behavior of Werkzeug. Werkzeug has a "reloader" process that monitors for file changes (and hence reloads if it detects a change in your .py files.
For more information on the reloader go here.
To get you through your trouble, I believe adding use_reloader=False to your call to app.run: app.run(use_reloader=False) would do the trick.
You can also see this SO answer for more information.
169
Gunicorn would always spawn at least 2 processes, even when you set --workers=1. One of the processes is a master process, which spawns other worker processes to handle requests.
38
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
