Apache Tomcat Exception - Too many open files

Question

We are running a web service in Apache Tomcat in Amazon Linux. Initially web-service is running properly. We are getting too many open files exception after making more than 1000 web request. Again this issue will be resolved when we re start the tomcat server.

Please find below the Exception

25-Apr-2016 10:05:52.628 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
 java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686)
        at java.lang.Thread.run(Thread.java:745)

PS : we are not doing any file related operations in the web service .

Answer 1

It looks like, that there is some limit on open files. As you are running on Linux I suspect you are running out of file descriptors.

Check out ulimit command to see the number of allowed opened files.

ulimit -n

You can change the number of open files by editing:

/etc/security/limits.conf

and adding something like this:

* soft nofile 4096
* hard nofile 4096

You can check more about limits.conf here.

The default limit is 1024 and can be too low for some Java applications.

More information about increasing the maximum number of open files in this article: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Answer 2

Although if "ulimit" is raised at some point down the line tomcat stops causing same error.

So in order to avoid this you can check list of open files for the application user on Linux using command "lsof -u username" or simply "lsof" and see if code related files are open ( eg..properties files ) if so kill those specific files using # kill -9 lsof -t -u username command for that specific tomcat user.

You need to fix your code to load those files writing simply in a static block of your classes. So that only one file loads even if multiple hits are made by any number of users.

Now you can re check after deploying new changes with the same lsof command and see. Only one file will be seen. This will permanently fix your issue without raising the ulimit each time

Answer 3

That is because socket connections are treated as files, so that means you have too many connections opened. Check the limitations (each OS has different policy about it - same goes for each server), how many ports you can open at same time, etc. You can use NIO to limit those things.