CannotGetJdbcConnectionException and StaleConnectionException problems

Question

I have recently run into a problem while trying to access my database through a java web application. I keep getting a CannotGetJdbcConnectionException along with a StaleConnectionException. I am using Websphere v8.5 as my local machine server. I have tried testing the Data Source Connection in the Administrative console and it always comes back successful. But whenever I try to access my application I get the above Exceptions. Does anyone have an idea of why this keeps happening and how I could possibly fix it? Below is the error message I keep getting.

I am using DB2 for my database

[10/1/15 7:47:23:550 CDT] 00000073 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on C:\Dev2\Utilities\IBM\AppServers\profiles\AppSrv03\logs\ffdc\server1_ef037f_15.10.01_07.47.23.5493440764669613130910.txt com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest() 309
[10/1/15 7:47:23:553 CDT] 00000073 webapp        E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[mvc-dispatcher]: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.ibm.websphere.ce.cm.StaleConnectionException: null DSRA0010E: SQL State = 56038, Error Code = -4,700
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:628)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:693)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:720)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:770)
    at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:190)
    at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:202)
    at com.shelter.trac.dao.AgencyDao.getStatesNDistricts(AgencyDao.java:32)
    at com.shelter.trac.serviceManager.AgencyManager.getStatesNDistricts(AgencyManager.java:27)
    at com.shelter.trac.controller.ApiController.getStatesNDistricts(ApiController.java:40)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
    at java.lang.reflect.Method.invoke(Method.java:613)
    at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:214)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:748)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:876)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:931)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:822)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:807)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1214)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456)
    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1027)
    at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3703)
    at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:962)
    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:522)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:311)
    at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1783)
Caused by: com.ibm.websphere.ce.cm.StaleConnectionException: null DSRA0010E: SQL State = 56038, Error Code = -4,700
    at com.ibm.ws.rsadapter.spi.ServerFunction.handleStaleStatement(ServerFunction.java:693)
    at com.ibm.ws.rsadapter.AdapterUtil.mapException(AdapterUtil.java:2277)
    at com.ibm.ws.rsadapter.AdapterUtil.translateSQLException(AdapterUtil.java:1582)
    at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getConnection(WSRdbDataSource.java:2243)
    at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.getConnection(WSManagedConnectionFactoryImpl.java:1761)
    at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1506)
    at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1096)
    at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2086)
    at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1766)
    at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3264)
    at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2567)
    at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1544)
    at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1027)
    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:642)
    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:609)
    at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
    ... 48 more

Answer 1

I did some more research on the subject especially the StaleConnectionException, and most of the article talked about the JDBC exceptions timing out or going stale. So I went into Websphere Administration Console and started looking around for properties about connections. In the Resources -> JDBC -> Data source -> [your Datasource] there is a page about "Websphere Application Server Data Source Propeties" and in it is a section called "Connection Validation Properties". I decided to give it a shot and checked the Validate New Connections checkbox and left the Number of Retries and Retry interval to their default settings and tested the app and it worked.