"ESAPI.properties could not be loaded by any means. fail." causing "Could not initialize class coldfusion.security.ESAPIUtils"

Question

I have two servers - one production and one development - running ColdFusion 9.0.1 on IIS 7.5 on Windows Server 2008 R2. The two are configured identically. We have a transient issue where, after weeks to months of uneventful uptime, some parts of the site (specifically CFIDE administrator portal and any page with a cfwindow tag) will start throwing "Could not initialize class coldfusion.security.ESAPIUtils" errors in the logs.

Based on some recommendations I uninstalled and reinstalled all of the hot fixes months ago, quintuple checking that I was applying them in the correct order and following the correct set of instructions.

That didn't fix things, but as I was combing through log files, I noticed that another ESAPI-related error ("ESAPI.properties could not be loaded by any means. fail.") appears in the logs after a jRun restart. I tried adding the following declaration to the java.args in jvm.config:

-Dorg.owasp.esapi.resources=E:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib

That seemed to fix the issue for several months; no errors, things worked fine. Then, yesterday the production server started throwing the errors again. I've tried restarting JRun and rebooting the server and the error persists. The development server is perfectly fine.

I tried creating a script that just instantiates and cfdumps an ESAPIUtils instance. On dev, it dumps the metadata about the object; on production, the page results in an error.

I've been fighting with this issue off and on for almost a year. Sometimes it resolves itself after several days, sometimes it continues for weeks. I've yet to figure out a way to "induce" the condition, so we're stuck with untestable "fixes" that seem to work for a while, then don't.

This seems completely tangential, but we've had instances where the built-in IsImageFile() function returns false for valid images. The IsImageFile() weirdness seems to start a bit before the "Could not initialize class coldfusion.security.ESAPIUtils" madness begins.

Below are the server versions:

ColdFusion Version: 9,0,1,274733
Operating System:   Windows Server 2008 R2 amd64 6.1
Web Server Software:    Microsoft-IIS/7.5
Java JVM:   1.8.0_05 Oracle Corporation 
JEE Server: JRun/4.0
Security Hotfixes (9.0.1):  APSB13-27, APSB13-19, APSB13-13, APSB13-10, ColdFusion 9.0.1     Cumulative Hotfix 4  (APSB13-03, APSB12-26, APSB12-21, APSB12-06, APSB11-29, APSB11-14, APSB11-04, APSB10-18), ColdFusion 9.0.1 Cumulative Hotfix 3, ColdFusion 9.0.1 Cumulative Hotfix 2, ColdFusion 9.0.1 Cumulative Hotfix 1
Connectors: JRun IIS 64 Bit Connector (Build 108858)

And a stack trace from cfusion-out.log:

08/27 11:37:52 Error [jrpp-58] - Could not initialize class     08/27 11:37:52 Error [jrpp-58] - Could not initialize class coldfusion.security.ESAPIUtils The specific sequence of files included or processed is: E:\web\cfadmin\webroot\CFIDE\administrator\index.cfm, line: 30
08/27 11:37:52 error ROOT CAUSE: 
java.lang.NoClassDefFoundError: Could not initialize class coldfusion.security.ESAPIUtils
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at coldfusion.runtime.java.JavaProxy.invoke(JavaProxy.java:97)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
at cflogin2ecfm1599616868.runPage(C:\work\ColdFusion\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\login.cfm:30)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
at cfApplication2ecfm1920815415._factor5(C:\work\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\Application.cfm:210)
at cfApplication2ecfm1920815415._factor9(C:\work\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\Application.cfm:202)
at cfApplication2ecfm1920815415.runPage(C:\work\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\Application.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.CfincludeFilter.include(CfincludeFilter.java:33)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:297)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:201)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
javax.servlet.ServletException: ROOT CAUSE: 
java.lang.NoClassDefFoundError: Could not initialize class coldfusion.security.ESAPIUtils
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at coldfusion.runtime.java.JavaProxy.invoke(JavaProxy.java:97)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
at cflogin2ecfm1599616868.runPage(C:\work\ColdFusion\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\login.cfm:30)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
at cfApplication2ecfm1920815415._factor5(C:\work\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\Application.cfm:210)
at cfApplication2ecfm1920815415._factor9(C:\work\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\Application.cfm:202)
at cfApplication2ecfm1920815415.runPage(C:\work\cf9_u1_final_hotfix\cfusion\wwwroot\CFIDE\administrator\Application.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.CfincludeFilter.include(CfincludeFilter.java:33)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:297)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:201)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Answer 1

Add a file read permission to the esapi.properties file present in <CF_HOME>\lib directory for each sandbox. If you can not access CF Admin console manually add this permission in neo-security.xml which can be found in <CF_HOME>\lib directory.