Menu
One of our customers has a newish problem: the application grinds to halt. Thread dump shows that all threads are hanging on network IO in JDBC calls.
We/I have never seen these 'network IO' hangs. Typically a slow machine w/ DB problems has either a) one or two long-running queries or b) some type of lock/deadlock. In either of these cases the threads 'hang' on different methods. I have never seen all 30+ threads hanging on network IO.
Below I have included an excerpt from the thread dump. All HTTP threads are hanging on the same java.net.SocketInputStream.read call.
I talked to their dba and sysadmin. According to them 'nothing has changed' in the environment recently which would cause this problem.
db environment
MSSQL 2005 64-bit Service Pack 2 Driver: sqljdbc.jar : 1.0 809 102
Note: they are running an older jdbc driver. AFAIK they tried upgrading from 1.0 to the 1.2 driver but had some other problem.
other environment issues
They're running both the app server and the db server in VMWare VM's. I don't know how this setup affects network performance.
Apparently this is the only application with this problem. I don't know anything else about their network architecture.
Questions * any insights on this problem? * if it is network, any next steps for analyzing?
Appendix A: Excerpt from Thread dump
All HTTP connections are hanging on the same method:
"TP-Processor31" daemon prio=5 tid=0x04085b78 nid=0x970 runnable [0x0764d000..0x0764fd6c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.microsoft.sqlserver.jdbc.DBComms.receive(Unknown Source)
at com.microsoft.sqlserver.jdbc.IOBuffer.sendCommand(Unknown Source)
- locked (a com.microsoft.sqlserver.jdbc.DBComms)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.sendExecute(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteQuery(Unknown Source)
421
Timeout SettingsThe default is 60 seconds. A value of zero forces caller to wait indefinitely. To improve performance set Max Wait Time to zero (0). This essentially blocks the caller thread until a connection becomes available.
If you don't close it, it leaks, and ties up server resources. @EJP The connection itself might be thread-safe (required by JDBC), but the applications use of the connection is probably not threadsafe. Think of things like different transaction isolation, boundaries (commit/rollback/autocommit) etc.
JDBC connection errors. Connection errors are typically caused by incorrect or invalid connection string parameters, such as an invalid host name or database name. Verify that you are using correct and valid connection string parameters for the Gate.
At the end of your JDBC program, it is required explicitly to close all the connections to the database to end each database session. However, if you forget, Java's garbage collector will close the connection when it cleans up stale objects.
We've had similar issues, and traced them back to a buggy JDK update (1.6.29).
We downloaded 1.6.27 (http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html#jdk-6u27-oth-JPR), re-set the JAVA_HOME environment and we were back on track.
163
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
