Ubuntu 16.04 systemd redis issues with ulimit

Question

I have been having issues with our new redis server after swapping from Ubuntu 14.04 to 16.04. The configuration of the open files limit using all the guides says to change the /etc/security/limits.conf with the following settings

*    soft nofile 65535
*    hard nofile 65535
root soft nofile 65535
root hard nofile 65535

And also to add to /etc/pam.d/common-session and /etc/pam.d/common-session-noninteractive

session required pam_limits.so

I have made all of these changes but redis is still starting up with a file limit of 4096 which under production load we quickly reach max connections.

Answer 1

After much searching I finally found that systemd has a standard limit of 4096 and regardless of what you set at the system level the systemd level will always take precedence. The fix for this is to change the systemd open file limit by editing /etc/systemd/system.conf and adding the following setting

DefaultLimitNOFILE=65536

It seems that there are a lot of default settings in systemd which override the system settings so have to be set in systemd

So if its any use for people these are my final settings for a Ubuntu 16.04 server for running production redis.

Edit /etc/systemd/system.conf (sudo nano /etc/systemd/system.conf) and add

DefaultLimitNOFILE=65536

Edit /etc/security/limits.conf (sudo nano /etc/security/limits.conf) and add

*    soft nofile 64000
*    hard nofile 64000
root soft nofile 64000
root hard nofile 64000

Edit /etc/pam.d/common-session (sudo nano /etc/pam.d/common-session) and add

session required pam_limits.so

Edit /etc/pam.d/common-session-noninteractive (sudo nano /etc/pam.d/common-session-noninteractive) and add

session required pam_limits.so

Edit /etc/rc.local (sudo nano /etc/rc.local) and add

sysctl -w net.core.somaxconn=65535

Edit /etc/sysctl.conf (sudo nano /etc/sysctl.conf) and add

vm.overcommit_memory = 1

Edit /etc/rc.local (sudo nano /etc/rc.local) and add

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Answer 2

The sysctl values net.core.somaxconn and vm.overcommit_memory are note related to ulimit or the number of open files. They are related to:

• net.core.somaxconn: will limit the number of opened but not accepted connections.
  https://serverfault.com/questions/518862/will-increasing-net-core-somaxconn-make-a-difference
• vm.overcommit_memory: the policy to overcommit memory when physical memory is exausted.
  https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

If your problem is with the limit of opened files (what ulimit -Sn/-Hn handles) then I would not suggest to play with this, unless you know exactly what you are doing.