MySQL Connection Timeout Issue - Grails Application on Tomcat using Hibernate and ORM

Question

I have a small grails application running on Tomcat in Ubuntu on a VPS. I use MySql as my datastore and everything works fine unless I leave the application for more than half a day (8 hours?). I did some searching and apparently this is the default wait_timeout in mysql.cnf so after 8 hours the connection will die but Tomcat won't know so when the next user tries to view the site they will see the connection failure error. Refreshing the page will fix this but I want to get rid of the error altogether. For my version of MySql (5.0.75) I have only my.cnf and it doesn't contain such a parameter, In any case changing this parameter doesn't solve the problem.

This Blog Post seems to be reporting a similar error but I still don't fully understand what I need to configure to get this fixed and also I am hoping that there is a simpler solution than another third party library. The machine I'm running on has 256MB ram and I'm trying to keep the number of programs/services running to a minimum.

Is there something I can configure in Grails / Tomcat / MySql to get this to go away?

Thanks in advance,

Gav

From my Catalina.out;

2010-04-29 21:26:25,946 [http-8080-2] ERROR util.JDBCExceptionReporter  - The last packet successfully received from the server was 102,906,722 milliseconds$
2010-04-29 21:26:25,994 [http-8080-2] ERROR errors.GrailsExceptionResolver  - Broken pipe
java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
         ...
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter  - Already closed.
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter  - Already closed.
2010-04-29 21:26:26,017 [http-8080-2] ERROR servlet.GrailsDispatcherServlet  - HandlerInterceptor.afterCompletion threw exception
org.hibernate.exception.GenericJDBCException: Cannot release connection
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException: Already closed.
        at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:84)
        at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
        ... 1 more

Answer 1

Referring to this article, you have stale connections in your DBCP connections pool that are silently dropped by OS or firewall.

The solution is to define a validation query and do a sanity check of the connection before you actually use it in your application. In grails this is actually done by modifying the grails-app/conf/spring/Resource.groovy file and add the following:

beans = {
  dataSource(BasicDataSource) {
    //run the evictor every 30 minutes and evict any connections older than 30 minutes.
    minEvictableIdleTimeMillis=1800000
    timeBetweenEvictionRunsMillis=1800000
    numTestsPerEvictionRun=3
    //test the connection while its idle, before borrow and return it
    testOnBorrow=true
    testWhileIdle=true
    testOnReturn=true
    validationQuery="SELECT 1"
  }
} 

Answer 2

In grails 1.3.X, you can modify the evictor values in the DataSource.groovy file to make sure pooled connections are used during idle. This will make sure the mysql server will not time out the connection.

production {
  dataSource {
    pooled = true
    // Other database parameters..
    properties {
       maxActive = 50
       maxIdle = 25
       minIdle = 5
       initialSize = 5
       minEvictableIdleTimeMillis = 1800000
       timeBetweenEvictionRunsMillis = 1800000
       maxWait = 10000
    }
}

A quick way to verify this works is to modify the MySQL my.cnf configuration file [mysql] element and add wait_time parameter with a low value.