I was having an issue where some sites were taking a long time to load (By "long time" I mean up to 16 seconds). Sometimes they might timeout entirely, which generated a Nginx 504 error. Usually, when a site timed out I could reload the site again and it would load quickly. The site that I was having issues with gets a very low amount of traffic. I'm testing the site by loading the Django admin index page in order to try and eliminate any slowness that could be caused because of poor code. It should also be noted that this particular site only uses the Django admin because it's an intranet-type site for staff only.
All the sites I'm hosting are on two Rackspace cloud servers. The first server is my app server, which has 1024 MB of RAM, and my second server is my database server, which has 2048 MB of RAM. The app server is serving up each site using Nginx, which serves all static files and proxies everything else to the Django Gunicorn workers for each site.
When looking at the database servers RAM and CPU load it seems like everything is fine on the database server.
$ free -m
total used free shared buffers cached
Mem: 1999 1597 402 0 200 1007
-/+ buffers/cache: 389 1610
Swap: 4094 0 4094
Top shows a CPU load average of: 0.00, 0.01, 0.05
In order to try and troubleshoot what is happening, I wrote a quick little script which prints out the memory usage on the app server.
Example print out with the site domains anonymized:
Celery: 23 MB
Gunicorn: 566 MB
Nginx: 8 MB
Redis: 684 KB
Other: 73 MB
total used free shared buffers cached
Mem: 993 906 87 0 19 62
-/+ buffers/cache: 824 169
Swap: 2047 828 1218
Gunicorn memory usage by webste:
site01.example.com 31 MB
site02.example.com 19 MB
site03.example.com 7 MB
site04.example.com 9 MB
site05.example.com 47 MB
site06.example.com 25 MB
site07.example.com 14 MB
site08.example.com 18 MB
site09.example.com 27 MB
site10.example.com 15 MB
site11.example.com 14 MB
site12.example.com 7 MB
site13.example.com 18 MB
site14.example.com 18 MB
site15.example.com 10 MB
site16.example.com 25 MB
site17.example.com 13 MB
site18.example.com 18 MB
site19.example.com 37 MB
site20.example.com 30 MB
site21.example.com 23 MB
site22.example.com 28 MB
site23.example.com 80 MB
site24.example.com 15 MB
site25.example.com 5 MB
Example Gunicorn config file:
pidfile = '/var/run/gunicorn_example.com.pid'
proc_name = 'example.com'
workers = 1
bind = 'unix:/tmp/gunicorn_example.com.sock'
Example Nginx config:
upstream example_app_server {
server unix:/tmp/gunicorn_example.com.sock fail_timeout=0;
}
server {
listen 80;
server_name example.com;
access_log /var/log/nginx/example.com.access.log;
error_log /var/log/nginx/example.com.error.log;
location = /favicon.ico {
return 404;
}
location /static/ {
root /srv/sites/example/;
}
location /media/ {
root /srv/sites/example/;
}
location / {
proxy_pass http://example_app_server;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
}
}
As you can see, there is a lot of memory that is swapped, so in order to fix my issues I upgraded the ram on the app server, which fixed the sites' slowness entirely. Even though I was able to fix the issue, it took me a lot longer than I would like and I still feel like I was basically guessing at what was causing the site slowness. All this leads me to my questions...
Gunicorn relies on the operating system to provide all of the load balancing when handling requests.
Nginx is a web server and reverse-proxy responsible for serving static content, gzip compression, ssl, proxy_buffers and other HTTP stuff while gunicorn is a Python HTTP server that interfaces with both nginx and your actual python web-app code to serve dynamic content.
Although there are many HTTP proxies available, we strongly advise that you use Nginx. If you choose another proxy server you need to make sure that it buffers slow clients when you use default Gunicorn workers. Without this buffering Gunicorn will be easily susceptible to denial-of-service attacks.
Green Unicorn, commonly shortened to "Gunicorn", is a Web Server Gateway Interface (WSGI) server implementation that is commonly used to run Python web applications.
That's a lot of sites to host on a server with only 1GB of RAM. You're at nearly 100% memory utilization, and the numbers you have are probably "standby" numbers. The RAM usage of each process can and will balloon in the process of serving requests. Right off the bat, you need to add more RAM to this instance and, better, move some of the sites off onto another server.
As to your questions:
Where'd you get the idea that sites become "inactive" and Gunicorn, then, has to load the site again? That's rubbish. As long as the Gunicorn process is running (i.e. not terminated manually or by an error on the site) it remains fully initialized and ready to go, whether it's been an hour or a month.
You're hacking at the leaves here, leaving the root untouched. There's nothing out of the ordinary with the memory usage of each Gunicorn process. It needs RAM to run. Your problem is trying to run too much on a severly underpowered server. No optimization is going to save you here. You need more RAM or more servers. Probably both.
No need. Again, the problem is already identified. Pretty clearly in fact by the numbers you posted.
There's no way to reliably know which processes are getting swapped. It changes every second and depends on which are actively running and need more RAM and which are inactive or simply not as active. When your server is this strapped for resources, it's spending half it's time just figuring out which process to juggle next, especially if they're all active and vying for resources.
Yes. Gunicorn recommends 2*cores+1. So on a dual-core system, that's 5; on a quad-core, 9. However, there's no way you could run even 5 workers for each of these sites on this one system. You can't even run 1 worker for each reliably.
It depends on the "things". But, when multiple sites are hosted on the same server, those servers are beasts spec-wise. On a small, probably VPS instance like you have, especially with only 1GB of RAM, one site is pretty much your limit. Two, maybe.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With