Handling Spring/Tomcat initialization failures

Question

Currently, if the database of our Spring app hosted on Tomcat is unavailable, context initialization fails and all requests return a 404.

What would be a good way to overcome this problem? Instead of the app being unavailable until the next Tomcat, I would like it to show an error message to the user while unavailable and automatically recover when the database is available (like it would if a database failure occurs while Tomcat is already running).

I can set all beans to lazy-init, but I'm not sure that's the best solution? Can't Tomcat retry the initialization every x seconds/requests and show a decent error page in the meantime? Any ideas on this?

An example of the errors thrown on startup when the DB is unavailable:

Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:401)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy43.getMetaData(Unknown Source)
at com.googlecode.flyway.core.dbsupport.DbSupportFactory.getDatabaseProductName(DbSupportFactory.java:103)
... 67 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 72 more
Mar 28, 2013 11:19:47 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Error listenerStart
Mar 28, 2013 11:19:47 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [] startup failed due to previous errors

Answer 1

From the exception stack trace I see that you are using c3p0 connection pool. Look at http://www.mchange.com/projects/c3p0/#configuring_recovery where you can configure your pool to retry for connection.

But this would keep the http thread active and imagine if you have high load your system will go down.

Its best to show an error page to the user. In this case i would show a 503 error. You can do this by putting Apache web server in front of your tomcat where you host the 503 page. There could be other ways to achieve this also.

Since your webapp itself will not start you cannot serve the error page from your webapp unless you configure it to be lazy init.

You can also try auto refresh on the error page every X seconds for the url user was trying to visit.