Menu
Earlier today I visited a site that I recently launched and the page was just filled with a big ugly jasperexception: java.io.filenotfoundexception: /opt/tomcat/webapps/root/web-inf/lib/jstl-1.2.jar (too many open files). I checked my server logs and tomcat logs and they were full of the same error. I increased tomcat's max open file limit as a temporary solution to keep the site running, but it's quickly filling up again. I've launched other sites with the same methods I used for this one, and never had this problem before. The only thing I did different this time was use Mobirise to make the base layout for the front-end, so I'm not ruling out that some scripts used by Mobirise could be the culprit. But when I get the list of files that tomcat has open, I see that it's got hundreds of copies of tomcat-users.xml open, and it just keeps opening more and more.
...
java 319 tomcat 935r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 936r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 937r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 938r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 939r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 940r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 941r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 942r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 943r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 944r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 945r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 946r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 947r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 948r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 949r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 950r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 951r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 952r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 953r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 954r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 955r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 956r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 957r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 958r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 959r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 960r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 961r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 962r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 963r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
java 319 tomcat 964r REG 253,1 2164 263714 /opt/tomcat/conf/tomcat-users.xml
...
I don't understand why it's doing this. I've been debugging, scanning through my code, and scouring the internet for issues where tomcat keeps opening too many files, but they all just say "increase the max file limit." None of them address the root of the problem or mention anything about tomcat just infinitely opening copies of a single file.
[EDIT]: Here is the full stack trace of the FileNotFoundException:
28-Nov-2018 03:51:33.857 SEVERE [http-nio-8080-exec-5] org.apache.catalina.core.ApplicationDispatcher.invoke Servlet.service() for servlet [jsp] threw exception
java.io.FileNotFoundException: /opt/tomcat/webapps/ROOT/WEB-INF/lib/jstl-1.2.jar (Too many open files)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:225)
at java.util.zip.ZipFile.<init>(ZipFile.java:155)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:130)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:164)
at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at org.apache.tomcat.util.descriptor.tld.TldResourcePath.openJar(TldResourcePath.java:134)
at org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:125)
at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:431)
at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:489)
at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445)
at org.apache.jasper.compiler.Parser.parse(Parser.java:144)
at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:206)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:385)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
at kom.kitten.gingerpage.servlets.HomeServlet.doGet(HomeServlet.java:16)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
28-Nov-2018 03:51:33.859 SEVERE [http-nio-8080-exec-5] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [kom.kitten.gingerpage.servlets.HomeServlet] in conte$
java.io.FileNotFoundException: /opt/tomcat/webapps/ROOT/WEB-INF/lib/jstl-1.2.jar (Too many open files)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:225)
at java.util.zip.ZipFile.<init>(ZipFile.java:155)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:130)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:164)
at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at org.apache.tomcat.util.descriptor.tld.TldResourcePath.openJar(TldResourcePath.java:134)
at org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:125)
at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:431)
at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:489)
at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445)
at org.apache.jasper.compiler.Parser.parse(Parser.java:144)
at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:206)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:385)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
at kom.kitten.gingerpage.servlets.HomeServlet.doGet(HomeServlet.java:16)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
This is the same stack trace that it logs every time the error occurs (i.e. every time someone attempts to load a page on the site while there are too many files open in tomcat)
230
This was caused by a bug in Tomcat 9.0.13 which has been fixed in Tomcat 9.0.14.
67
I just ran into the same issue after moving an application from a tomcat7 instance to a tomcat 9 instance. I have a feeling this may be a tomcat bug. I removed the tag
<Resource pathname="conf/tomcat-users.xml"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
type="org.apache.catalina.UserDatabase"
auth="Container" name="UserDatabase"/>
in the GlobalNamingResources tag in $CATALINA_HOME/conf/server.xml and after a restart the issue resolved itself. So I'm guessing the issue lies with Tomcat's management app.
23
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
