what is java.io.EOFException, Message: Can not read response from server. Expected to read 4 bytes, read 0 bytes

Question

This question has been asked a couple of times in SO and many times in other sites. But I didn't get any satisfiable answer.

My problem:
I have a java web application which uses simple JDBC to connect to mysql database through Glassfish application server.

I have used connection pooling in glassfish server with the following configurations:
Initial Pool Size: 25
Maximum Pool Size: 100
Pool Resize Quantity: 2
Idle Timeout: 300 seconds
Max Wait Time: 60,000 milliseconds

The application has been deployed for last 3 months and it was running flawlessly too.
But from last 2 days the following error is coming at the time of login.

Partial StackTrace

com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: No operations allowed after connection closed.Connection was implicitly closed due to underlying exception/error:  

** BEGIN NESTED EXCEPTION **  

com.mysql.jdbc.CommunicationsException  
MESSAGE: Communications link failure due to underlying exception:  

** BEGIN NESTED EXCEPTION **  

java.io.EOFException  
MESSAGE: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.  

STACKTRACE:  

java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.  
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1997)  
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)  
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)  
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)  
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)  
at com.mysql.jdbc.Connection.execSQL(Connection.java:3256)  
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313)  
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1448)  
............  
............  
my application traces....  

What caused this error suddenly ? I have lost a lot of time for this.

EDIT : The problem even persists after restarting the server. As per DBA two of the important mysql server configurations are:
wait_timeout : 1800 seconds
connect_timeout : 10 seconds
NOTE : Other applications deployed in the same server connecting to the same database and using different pools are running smoothly.

EDIT-2 : After reading a lot of things and expecting some positive outcome I made these changes to my connection pool.

Max Wait Time : 0 (previously it was 60 seconds)
Connection Validation : Required
Validation Method : table
Table Name : Demo
Validate Atmost Once : 40 seconds
Creation Retry Attempts : 1
Retry Intervals : 5 seconds
Max Connection Usage : 5

And this worked as the application is running for 3 days consistently. But I got a very strange and interesting result of out this. While monitoring the connection pool, I found these figures:

NumConnAcquired : 44919 Count
NumConnReleased : 44919 Count
NumConnCreated : 9748 Count
NumConnDestroyed : 9793 Count
NumConnFailedValidation : 70 Count
NumConnFree : 161 Count
NumConnUsed : -136 Count

How can the NumConnFree become 161 as I have Maximum Pool Size = 100 ?
How can the NumConnUsed become -136, a negative number ?
How can the NumConnDestroyed > NumConnCreated ?

Answer 1

The connection has failed, possibly due to a firewall idle-timeout, etc. If you don't have your JDBC driver configured to reconnect on failure, then this error will not go away unless you open a new connection.

If you are using a database connection pool (you are using one, right?), then you probably want to enable it's connection-checking features like issuing a query to check to see if the connection is working before handing it back to the application. In Apache commons-dbcp, this is called the validationQuery and is often set to something simple like SELECT 1.

Since you are using MySQL, you ought to use a Connector/J-specific "ping" query that is lighter-weight than actually issuing a true SQL query and set your validation query to /* ping */ SELECT 1 (the ping part needs to be exact).

Answer 2

This is EndOfFileException in java, happens when your cursor start point is on the end point or when database connection closed because of some unexpected exception.

Answer 3

It most likely means that the database has restarted or the network connection to the database has been broken (e.g. a NAT connection has timed out) ... and your webapp is trying to use a stale database connection.

If the problem persists after restarting the web container, it could be something more serious.

---

You asked the following:

How can the NumConnFree become 161 as I have Maximum Pool Size = 100 ?
How can the NumConnUsed become -136, a negative number ?
How can the NumConnDestroyed > NumConnCreated ? 

On the face of it, these don't make sense. However, they could simply be the result of some usage counters being updated in a non-thread-safe way. This is not necessarily related to your original problem.

Answer 4

While I don't have definitive solutions, it seems something is interfering communication between the app server and db. Following are few things you can try to isolate the problems:

• Try to determine wether this is mysql problem or java code problem. Try connect to mysql using the command line tool from the same host as the app server and issue similar SQL to perform login. Test using a simple java code that does a select, deploy it to the same infrastructure, see what happen etc. Also check the mysql server log, see if you can find anything useful

• There are two way an idle connection get closed: by the connection pool code that runs inside app server, or by mysql itself. Make sure you check the config on both sides

• Check if any network infrastructure config has changed recently. Was there any new firewall rule in place interfering with app server <--> mysql connectivity? Was there any settings that prohibits open TCP connection idling longer than X?

• Try a different connection pooling library just to eliminate the possibility it's the connection pooling

Good luck