WiredTiger - “too many open files error” during resync of a secondary replica set member

Question

I'm upgrading a secondary replica set member to wiredTiger. I have upgraded it from MongoDB 2.6.3 to 3.0.4 and changed the storage engine to wiredTiger. Now it is resyncing all data from the primary. At some point the following error is received, and the process starts all over again:

2015-07-22T13:18:55.658+0000 I INDEX [rsSync] building index using bulk method

2015-07-22T13:18:55.664+0000 I INDEX [rsSync] build index done. scanned 1591 total records. 0 secs

2015-07-22T13:18:56.397+0000 E STORAGE [rsSync] WiredTiger (24) [1437571136:397083][20413:0x7f3d9ed29700], file:WiredTiger.wt, session.create: WiredTiger.turtle: fopen: Too many open files

2015-07-22T13:18:56.463+0000 E REPL [rsSync] 8 24: Too many open files

2015-07-22T13:18:56.463+0000 E REPL [rsSync] initial sync attempt failed, 9 attempts remaining

The same machine was previously running 2.6.3 version without any open file limits issues. I'm aware that wiredTiger might be creating much more files, so it must be it, but does it keep them all open simultaneously ?

For reference:

cat /proc/sys/fs/file-max

10747371

In /etc/init.d/mongod the configuration is:

ulimit -n 64000

According to the documentation it seems that mongo holds a file descriptor for every data file. As in wiredTiger this results in a file for each collection + a file for each index, according to a calculation for our usecase, can add up to over 700K.

So I can change the ulimit to 700000 or higher, but I'm wondering whether this is the most correct solution, and what alternatives exist.

Answer 1

WiredTiger will clean up open file descriptors based on how long they have been idle, but during heavy activity across a large number of collections and indexes you are going to end up bounded by the ulimit on open files.

So, yes, basically you need to increase the limit until you no longer hit the problem, stick with MMAPv1, or consolidate some collections. I would also recommend filing a feature request outlining your use case with your sample numbers for a way to prevent this type of pattern (more than one collection per file for example).