Java file limit on OSX lower than in bash

Question

I've increased the max files limit on my macbook pro so that Elasticsearch can work with more files, but it isn't working.

I run the command 'ulimit -a' and it says "open files" is 100,000. I can run a simple shell script like this:

export counter=0
while (true) ; do touch "/tmp/foo${counter}" ; export counter=`expr $counter + 1` ; done

And I'm able to create lots of files (over 60,000 before I killed the script).

However, using Java code to create RandomAccessFiles in an empty sub-directory of the "/tmp" directory, I can only make 10,232 files before I get the error: java.io.FileNotFoundException (Too many open files). Here's my Java code:

import java.io.*;
import java.util.*;

public class max_open_files {
    public static void main(String ... args) throws Exception {
        File testDir = new File("/tmp/tempsubdir");
        testDir.mkdirs();

        List<File> files = new LinkedList<File>();
        List<RandomAccessFile> fileHandles = new LinkedList<RandomAccessFile>();

        try {
            while (true) {
                File f = new File(testDir, "tmp" + fileHandles.size());
                RandomAccessFile raf = new RandomAccessFile(f, "rw");
                files.add(f);
                fileHandles.add(raf);
            }
        } catch (Exception ex) {
            System.out.println(ex.getClass() + " " + ex.getMessage());
        }

        for (RandomAccessFile raf : fileHandles) raf.close()

        for (File f : files) f.delete();

        System.out.println("max open files: " + fileHandles.size());
    }
}

This java code is similar to the code in Elasticsearch that tests the limit on the number of files (in the FileSystemUtils class in the maxOpenFiles method). So Elasticsearch has the same problem my Java program has.

Why can the shell script make so many more files than the Java code (mine and Elasticsearch's)? Why does the high system limit on the number of files not get recognized by the Java code?

Update May 13 4:50pm CDT: I created a C version of the test program to see if the limit was Java-specific and it appears to be so. The C version below can open 32,765 files while the Java code is limited to 10,232 files.

#include <stdio.h>
#include <stdlib.h>

int main(int argc, const char *argv) {
    char name[100];
    FILE *fp;
    int ndx;

    for (ndx = 0; ndx < 50000; ndx++) {
        sprintf(name, "/tmp/foo%d.txt", ndx);
        fp = fopen(name, "w");
        if (fp == NULL) {
            fprintf(stdout, "Can not create file %d\n", ndx);
            return 1;
        }
        fprintf(fp, "hello %d", ndx);
    }

    return 0;
}

Answer 1

Java Virtual Machine Option Reference for Mac

-XX:- MaxFDLimit

Directs the VM to refrain from setting the file descriptor limit to the default maximum. The default behavior is to set the limit to the value specified by OPEN_MAX, which is 10240. Normally, this is the maximum number of files that a process may have open. It is possible, however, to increase this limit to a user-specified value with the sysctl utility. Under such circumstances, you may want to pass -XX:-MaxFDLimit to stop the Java VM from restricting the number of open files to 10240.