"Too many open files" in a grails application - how do I close opened files and streams correctly?

Question

I am developing a rather complex web application that invokes several externally executed processes, a grails background process and reads/write from/to several files - all in one controller. All was fine until I tested it with many requests in close time proximity. When I do this, I get the following java error message in my tomcat catalina log-file:

WARNING: Exception executing accept
java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:312)
    at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:666)
    at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:877)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
    at java.lang.Thread.run(Thread.java:662)

At first, after some happy googeling and reading, I suspected that it might be a 'system problem', i.e. that I have to raise the limit for open files for the user that executes tomcat - but that is not the case.

Then I started to read a lot of advice on Java, not on Grails & Groovy, because if you google for this problem with Grails, you don't find so much. I now suspect that my problem is caused by "too many open streams" (or something like that) instead of too many actually open files (since the number of open files is really not THAT big).

I have a lot of operations of the following four types in one closure:

1) Opening files and writing to them:

def someFile = new File("/some/file.txt")
someFile << "Some content\n"

2) Executing commands:

def cmd = "bash some-cmd".execute()
cmd.waitFor()

3) Reading content from files:

def fileContent = new File("/some/file.txt").text

4) Reading content from files on the web:

def URL url = new URL("http://www.some.link");
def URLConnection uc = url.openConnection()
def BufferedReader br = new BufferedReader(new InputStreamReader(uc.getInputStream()))
...
br.close()

As you can see, the only thing that I explicitly close is the BufferedReader with the InputStream, I believe br.close() closes them both.

Do I have to close any of the other opened connections, or better: can I do that? What would be the command to do this? Or do you think my problem is really not caused by a "forgotten, open stream"?

My question mainly originates in the answers to Why do I get "Too many open files" errors? and IOException: Too many open files .

I am using grails 1.1.1 (I KNOW that it is outdated but I had serious problems migrating my application to the current version and I gave up on it after many hours of work), groovy 1.8.0, tomcat 6.0.28, apache 2.2.16 on ubuntu 10.10.

The answer to solve my "Too many open files" problem is very related to Stephen C's answer. It seems like the first cause of my error was indeed the not closed BufferedReader Stream. I basically transferred his java-code suggestion directly to grails:

def URL url = new URL("http://www.some.link");
def URLConnection uc = url.openConnection()
def BufferedReader br = new BufferedReader(new InputStreamReader(uc.getInputStream()))
try{
    ...
}finally{
    br.close()
}

-> This definitely solved the problem with "Too many open files" and I am now even able to see what was the real source of the problem.

Answer 1

As you can see, the only thing that I explicitely close is the BufferedReader with the InputStream, I believe br.close() closes them both.

It does ... but only if it is executed.

I'm not a groovy / grails programmer, but in Java there is a common mistake that people make that can result in file descriptor leaks. For example,

InputStream is = new FileInputStream(someFile);

// do some work
...

is.close();

The problem is that the statements indicated by the ellipsis (...) may throw an exception. If they do that, the is.close() call doesn't happen, and a file descriptor is leaked. The solution (in Java) is to write the code like this:

InputStream is = new FileInputStream(someFile);
try {
   // do some work
   ...
} finally {
   is.close();
}

or (in Java 7) as:

try (InputStream is = new FileInputStream(someFile)) {
   // do some work
   ...
}