Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

IOException: Too many open files

I'm trying to debug a file descriptor leak in a Java webapp running in Jetty 7.0.1 on Linux.

The app had been happily running for a month or so when requests started to fail due to too many open files, and Jetty had to be restarted.

java.io.IOException: Cannot run program [external program]: java.io.IOException: error=24, Too many open files     at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)     at java.lang.Runtime.exec(Runtime.java:593)     at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)     at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:246) 

At first I thought the issue was with the code that launches the external program, but it's using commons-exec and I don't see anything wrong with it:

CommandLine command = new CommandLine("/path/to/command")     .addArgument("..."); ByteArrayOutputStream errorBuffer = new ByteArrayOutputStream(); Executor executor = new DefaultExecutor(); executor.setWatchdog(new ExecuteWatchdog(PROCESS_TIMEOUT)); executor.setStreamHandler(new PumpStreamHandler(null, errorBuffer)); try {     executor.execute(command); } catch (ExecuteException executeException) {     if (executeException.getExitValue() == EXIT_CODE_TIMEOUT) {         throw new MyCommandException("timeout");     } else {         throw new MyCommandException(errorBuffer.toString("UTF-8"));     } } 

Listing open files on the server I can see a high number of FIFOs:

# lsof -u jetty ... java    524 jetty  218w  FIFO        0,6      0t0 19404236 pipe java    524 jetty  219r  FIFO        0,6      0t0 19404008 pipe java    524 jetty  220r  FIFO        0,6      0t0 19404237 pipe java    524 jetty  222r  FIFO        0,6      0t0 19404238 pipe 

when Jetty starts there are just 10 FIFOs, after a few days there are hundreds of them.

I know it's a bit vague at this stage, but do you have any suggestions on where to look next, or how to get more detailed info about those file descriptors?

like image 968
Mirko N. Avatar asked Jan 11 '10 20:01

Mirko N.


People also ask

What causes too many open files?

"Too many open files " errors happen when a process needs to open more files than it is allowed by the operating system. This number is controlled by the maximum number of file descriptors the process has. 2. Explicitly set the number of file descriptors using the ulimit command.

How do I close open files in Linux?

They will disappear as your typing extends down the screen. To close a file to which no changes have been made, hit ESC (the Esc key, which is located in the upper left hand corner of the keyboard), then type :q (a colon followed by a lower case "q") and finally press ENTER.

What is Java net SocketException too many open files?

You can see that, open files (-n) 256, which means only 256 open file handles per process is allowed. If your Java program, remember Tomcat, weblogic or any other application server are Java programs and they run on JVM, exceeds this limit, it will throw java. net. SocketException: Too many files open error.


1 Answers

The problem comes from your Java application (or a library you are using).

First, you should read the entire outputs (Google for StreamGobbler), and pronto!

Javadoc says:

The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

Secondly, waitFor() your process to terminate. You then should close the input, output and error streams.

Finally destroy() your Process.

My sources:

  • http://stuffthathappens.com/blog/2007/11/28/crash-boom-too-many-open-files/
  • http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4
  • http://kylecartmell.com/?p=9
like image 58
ofavre Avatar answered Sep 25 '22 12:09

ofavre