Nginx and sysctl configuration - Performance setting

Question

Nginx is acting as a reverse proxy for adserver, receiving 20k requests per minute. Response happens within 100ms from the adserver to the nginx

Running on a Virtual Machine with configuration as 128GB RAM 4 vCPU 100GB HDD

Considering above, what is good setting of Nginx and also sysctl.conf

Answer 1

Please keep in mind that kernel tuning is complex and requires a lot of evaluation until you get the correct results. If someone spots a mistake please let me know so that I can adjust my own configuration :-)

Also, your memory is quite high for the amount of requests if this server is only running Nginx, you could check how much you are using during peak hours and adjust accordingly.

An important thing to check is the amount of file descriptors, in your situation I would set it to 65.000 to cope with the 20.000+ requests per second. The reason is that in a normal situation you would only require about 4.000 file descriptors as you have 4.000 simultanious open connections (20.000 * 2 * 0.1). However in case of an issue with a back end it could take up to 1 second or more to load an advertisement. In that case the amount of simultanious open connections would be higher:

20.000 * 2 * 1.5 = 60.000. 

So setting it to 65K would in my opinion be a save value.

You can check the amount of file descriptors via:

cat /proc/sys/fs/file-max

If this is below the 65000 you'll need to set this in the /etc/sysctl.conf:

fs.file-max = 65000

Also for Nginx you'll need to add the following in the file: /etc/systemd/system/nginx.service.d/override.conf

[Service]
LimitNOFILE=65000

In the nginx.conf file:

worker_rlimit_nofile 65000;

When added you will need to apply the changes:

sudo sysctl -p
sudo systemctl daemon-reload
sudo systemctl restart nginx

After these settings the following settings will get you started:

vm.swappiness = 0   # The kernel will swap only to avoid an out of memory condition
vm.min_free_kbytes = 327680 # The kernel will start swapping when memory is below this limit (300MB)

vm.vfs_cache_pressure = 125 # reclaim memory which is used for caching of VFS caches quickly
vm.dirty_ratio = 15 # Write pages to disk when 15% of memory is dirty
vm.dirty_background_ratio = 10 # System can start writing pages to disk when 15% of memory is dirty

Additionally I use the following security settings in my sysctl configuration in conjunction with the tunables above. Feel free to use them, for credits

# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Turn on syncookies for SYN flood attack protection
net.ipv4.tcp_syncookies = 1
# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
# No source routed packets here
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Make sure no one can alter the routing tables
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
# Don't act as a router
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
# Turn on execshild
kernel.exec-shield = 1
kernel.randomize_va_space = 1

As you are proxying request I would add the following to your sysctl.conf file to make sure you are not running out of ports, it is optional but if you are running into issues it is something to keep in mind:

net.ipv4.ip_local_port_range=1024 65000

As I normally evaluate the default settings and adjust accordingly I did not supply the IPv4 and ipv4.tcp_ options. You can find an example below but please do not copy and paste, you'll be required to do some reading before you start tuning these variables.

# Increase TCP max buffer size setable using setsockopt()
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
# Increase Linux auto tuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
# Tcp Windows etc
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1

The above parameters is not everything you should consider, there are many more parameters that you can tune, for example:

• Set the amount of worker processes to 4 (one per CPU core).
• Tune the backlog queue.
• If you do not need an acccess log I would simply turn it off to remove the disk I/O.
• Optionally: lower or disable gzip compression if your CPU usage is getting to high.