Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Too many open files: how many are open, what they are, and how many can the JVM open

Tags:

java

jvm

I'm getting this exception in Java:

java.io.FileNotFoundException: (Too many open files)  

I'm looking for the ways to eliminate this problem.

This error obviously indicates that JVM has allocated too many handles and underlying OS won't let it have more. Either I've got leak somewhere with improperly closed connections/streams.

This process runs for days non-stop and eventually throws the exception. It repeatedly happens after 12-14 days of up-time.

How do you fight this? Is there a way to get a list of allocated handles in JVM or track when it hits certain amount? I'd love to have them printed and see how it grows and when. I can't use a profiler because it's a production system and have difficulties to reproduce it in development. Any suggestion?

I am monitoring free heap size and raising an "alarm" when it approaches 1% of the total specified in -Xmx. I also know that if my thread count hits above 500, then something definitely goes out of hand. Now, is there a way to know that my JVM allocates too many handles from OS and doesn't give them back, e.g. sockets, opened files, etc. If I'd knew that, I'd know where to look and when.

like image 443
Dima Avatar asked Feb 16 '10 12:02

Dima


People also ask

How do you handle too many open files?

The Too many open files message occurs on UNIX and Linux operating systems. The default setting for the maximum number of open files might be too low. To avoid this condition, increase the maximum open files to 8000 : Edit the /etc/security/limit.

How many files can a process open?

Linux systems limit the number of file descriptors that any one process may open to 1024 per process. (This condition is not a problem on Solaris machines, x86, x64, or SPARC). After the directory server has exceeded the file descriptor limit of 1024 per process, any new process and worker threads will be blocked.

Why are there so many open files?

The "Too many open files" message means that the operating system has reached the maximum "open files" limit and will not allow SecureTransport, or any other running applications to open any more files. The open file limit can be viewed with the ulimit command: The ulimit -aS command displays the current limit.


2 Answers

Since you are on Linux, I'd suggest, that you check the /proc-Filesystem. Inside proc, you will find a folder with the PID of your process containing a folder calld 'fd'. If your process id is 1234, the path is be

/proc/1234/fd 

Inside that folder, you will find links to all opened files (do a 'ls -l'). Usually, you can tell by the filename which library / code might open and not close the file.

like image 22
phisch Avatar answered Sep 29 '22 01:09

phisch


You didn't say which OS you are running on, but if you are running on Linux you can use the lsof command

lsof -p <pid of jvm> 

That will list all the files opened by the JVM. Or if you are running on Windows you can Process Explorer which will show all the open files for all the processes.

Doing this will hopefully allow you to narrow down which bit of the code is keeping the files open.

like image 67
bramp Avatar answered Sep 29 '22 00:09

bramp