emfile error running CouchDB on Ubuntu 16

Question

I recently upgraded to Ubuntu 16 and CouchDB 1.6. When my server gets hammered with a ton of connections, I get emfile errors in the couch.log. I've seen these errors before and they appear to be related to the System Resource Limits mentioned at https://wiki.apache.org/couchdb/Performance. I believe my main problem is with ERL_MAX_PORTS. I've already increased my ulimit to 40000 (via PAM) and added export ERL_MAX_PORTS=40000 to /etc/default/couchdb.

Ubuntu 16 uses systemd so I added Environment="ERL_MAX_PORTS=40000" to the [Service] section in /etc/systemd/system/multi-user.target.wants/couchdb.service, but this does not seem to work.

I even tried modifying /usr/bin/couchdb and adding env ERL_MAX_PORTS=40000 to the top of the script.

The only thing that appears to work is when I manually run couchdb from the command line, e.g. $ env ERL_MAX_PORTS=40000 && sudo /usr/bin/couchdb. This is not an ideal solution and I would rather get this working with systemd.

I was previously running CouchDB 1.5 on Ubuntu 14 and increased my System Resource Limits (see https://wiki.apache.org/couchdb/Performance) to about 40000 and it was handling many connections without any issues.

Has anyone had any luck solving this problem on Ubuntu 16?

Answer 1

I just stumbled upon https://stackoverflow.com/a/39506150/2831606 and it was the missing nugget that I needed to get this working.

Specifically, I had to modify /etc/systemd/system.conf and add

DefaultLimitNOFILE=40000

So, to summarize, here are my configurations:

Let 100-couchdb.conf be:

#<domain>    <type>    <item>    <value>
couchdb      hard      nofile    40000
couchdb      soft      nofile    40000
*            hard      nofile    40000
*            soft      nofile    40000
root         hard      nofile    40000
root         soft      nofile    40000

1) Set up limits via PAM:

cp 100-couchdb.conf /etc/security/limits.d/100-couchdb.conf
cp 100-couchdb.conf /etc/security/limits.conf
echo "session    required   pam_limits.so" >> /etc/pam.d/su

2) Increase the allowable number of files in systemd

echo "DefaultLimitNOFILE=40000" >> /etc/systemd/system.conf

3) Increase the number of connections permitted by Erlang via systemd

sed -i "s/\[Service\]/\[Service\]\nEnvironment=ERL_MAX_PORTS=40000/g" /etc/systemd/system/multi-user.target.wants/couchdb.service

4) Also increase the number of connections permitted by Erlang (I'm not sure if this is needed as we already configured this via systemd, but it doesn't seem to hurt)

echo "export ERL_MAX_PORTS=40000" >> /etc/default/couchdb

Note: while running some stress tests, I found that I needed to increase the limits to about 40000 to support 10,000 concurrent connections. I'm guessing this is due to some overhead where multiple files are needed per connection.