In Java: "Too many open files" error when reading from a network path

Question

I have the code below, which simply reads all the files from a folder. There are 20,000 files in this folder. The code works good on a local folder (d:/files), but fails on a network path (//robot/files) after reading about 1,000 - 2,000 files.

Update: the folders are copies of each other.

What causes this problem and how to fix it?

package cef_debug;

import java.io.*;

public class Main {

    public static void main(String[] args) throws Throwable {
        String folder = args[0];
        File[] files = (new File(folder)).listFiles();
        String line;
        for (int i = 0; i < files.length; i++) {
            BufferedReader br = new BufferedReader(new FileReader(files[i]));
            while ((line = br.readLine()) != null) {
            }
            br.close();
        }
    }
}

I get the following error when reading from a network path (//robot/files):

Exception in thread "main" java.io.IOException: Too many open files
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:106)
        at java.io.FileReader.<init>(FileReader.java:55)
        at cef_debug.Main.main(Main.java:12)
Java Result: 1

Line 12 is the line:

BufferedReader br = new BufferedReader(new FileReader(files[i]));

Answer 1

There is a documented bug for some java versions and some file opens to hit a limit of 2035. It is possible that you might've just hit that.

From the comments:

To clarify the issue, on win32 system there are three ways to open a file:

1: Using Win32 API

2: Using MFC class framework lib.

3: using C-Library API (open() and fopen())

Other than the third option, i.e. option 1 and 2 have practically no limitation in opening number of files. The third method is restricted (for the reason not known to me) to open only approx. 2035 files. That is why MS JVM is able to open unlimited (practically) files, but SUN JVM fails after 2035 files (my guess is it is using 3rd method to open file).

Now, this is an old issue fixed quite some time ago, but it is possible that they would be using the same function on network access, where the bug could still exist.

Even without closing the handle or the stream, windows should be able to open >10000 file handles and keep them open, as demonstrated by this test code in the bug comments:

import java.util.*;
import java.io.*;

// if run with "java maxfiles 10000", will create 10k files in the current folder
public class maxfiles
{
    static int count = 0;
    static List files = new ArrayList();

    public static void main(String args[]) throws Exception
    {
        for (int n = 0; n < Integer.parseInt(args[0]); n++) {
            File f = new File("file" + count++);
            //save ref, so not gc'ed
            files.add(new PrintStream(new FileOutputStream(f)));
        }
        Iterator it = files.iterator();
        while (it.hasNext()) {
            PrintStream out = ( PrintStream) it.next();
            out.println("foo");
            out.flush();
        }
        System.out.println("current files open: " + files.size());
    } //~main
}

You could test running it on the network share, and report a bug if it fails. You could also try with a different JDK. At least with OpenJDK source, I couldn't see any other calls except WinAPI calls, so I'd try if the behaviour is the same.

Answer 2

Try:

package cef_debug;

import java.io.*;

public class Main {

    public static void main(String[] args) throws Throwable {
        String folder = args[0];
        File[] files = (new File(folder)).listFiles();
        String line;
        for (int i = 0; i < files.length; i++) {
            try {
                BufferedReader br = new BufferedReader(new FileReader(files[i]));
                while ((line = br.readLine()) != null) {
                }
            } finally {
                try {
                    if (br != null){
                        br.close();
                    }
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}