Hibernate "APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!"

Question

I recently got the following messages in our logs followed by a JVM crash (Due to OOME). I am not sure what to make of this and would really appreciate any guidance.

2015-03-19 21:15:02,457 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 
2015-03-19 21:26:29,543 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 3
    Active Threads: 3
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19397937
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5c7d3838
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7aea62dd
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@55622ff2
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@74004a8
Pool thread stack traces:
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
        java.net.SocketOutputStream.socketWrite0(Native Method)
        java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
        com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
        com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1751)
        com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3425)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2196)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:718)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:408)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:282)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
        com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)


2015-03-19 21:56:59,137 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 
2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.java [process]:566) - Java heap space 
java.lang.OutOfMemoryError: Java heap space
    at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1269)
    at org.exist.storage.btree.BTree$BTreeNode.access$16(BTree.java:1239)
    at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:460)
    at org.exist.storage.btree.BTree.scanSequential(BTree.java:413)
    at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:2039)
    at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1835)
    at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
    at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
    at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
    at org.exist.storage.btree.BTree$BTreeNode.access$12(BTree.java:1734)
    at org.exist.storage.btree.BTree.query(BTree.java:379)
    at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.java:259)
    at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.java:457)
    at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.java:585)
    at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.java:860)
    at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.java:1)
    at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.java:162)
    at org.exist.xquery.LocationStep.getAttributes(LocationStep.java:645)
    at org.exist.xquery.LocationStep.eval(LocationStep.java:434)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
    at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:446)
    at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326)
    at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:251)
    at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:238)
    at org.exist.xquery.LocationStep.eval(LocationStep.java:462)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:142)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
    at org.exist.xquery.BindingExpression.eval(BindingExpression.java:164)
2015-03-19 21:56:59,147 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 3
    Active Threads: 3
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3a9d08ca
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ecdd11
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@44ff846d
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5ce5850a
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@eec1d04
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6b8d4d9d
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@53e9706d
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@23d472cf
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4dbe4f8c
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4c5e0203
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@54ac79fd
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@546e2bad
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6b13cc83
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@57e185f8
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@60357d68
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@45231180
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@3021aa73
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6bb437ca
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@2021c9e9
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@7d53637c
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@409c2c97
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@adc5929
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@241ca71a
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@42b26866
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@636b1c33
        com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@b160466
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4af34669
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b53e609
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2062ebd4
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b6cfe8a
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4b7c2380
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f9be748
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78108924
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@474b002
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2ebee32f
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e0fe017
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42aa175b
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@637f5bac
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3a017b77
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b4f2b78
Pool thread stack traces:
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:408)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:370)
        com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2532)
        com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
        com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
        com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
        com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
        com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
        com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
        com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
        com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
        com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2596)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
        com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
        com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
        com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
        com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
        com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
        com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
        java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
        java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
        com.newrelic.agent.TransactionService.addTransaction(TransactionService.java:142)
        com.newrelic.agent.Transaction.getTransaction(Transaction.java:1104)
        com.newrelic.agent.Transaction.getTransaction(Transaction.java:1087)
        com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.java:136)
        com.newrelic.agent.TracerService.getTracer(TracerService.java:41)
        com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.java:55)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java)
        com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
        com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
        com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
        com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
        com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
        com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

We are using hiberanate and c3p0 with the following c3p0 configuration:

We are using the following maven artifacts for hibernate and c3p0:

 <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-core</artifactId>
            <version>4.3.6.Final</version>
        </dependency>

        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-c3p0</artifactId>
            <version>4.3.6.Final</version>
        </dependency>

with the following c3p0 configuration:

  configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
  .setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
  .setProperty("hibernate.c3p0.idle_test_period", "1000")
  .setProperty("hibernate.c3p0.min_size", "20")
  .setProperty("hibernate.c3p0.max_size", "50")
  .setProperty("hibernate.c3p0.timeout", "1800")
  .setProperty("hibernate.c3p0.max_statements", "50")

The server is under very light load ~5 queries a second. (Java 8)

Answer 1

I stumbled into the same exception and the reason was a wrong password of the db user...

c3p0 you're a funny guy

Answer 2

So, directly the issue is that the Connection pool was trying to acquire new Connections, but the tasks attempting to acquire those tasks froze for a long period of time, so long that c3p0 decided the tasks must be deadlocked and then discarded and replaced the Thread pool. Later, the Thread pool was hung on idle connection test tasks.

Normally, "hung" tasks tend to look like the second Thread underneath the first "Pool thread stack traces:" label above: performing network IO that fails to complete. Your circumstances are odd in that two of the three threads are not stuck in IO. They have barely begun to do anything, yet they aren't live. Then you experience an OutOfMemoryError, and you get another APPARENT DEADLOCK on idle connection test tasks that also look like they mostly should be live.

Maybe your application is very close to some kind of resource limits that are causing things to run very sluggishly? Straightforwardly, you might increase the amount of memory available to this application (or modify it to have a lower memory footprint). You experience an OOME the second time the Thread pool tries to flush and recreate hung Threads, not directly provoked by that, but quite possibly caused by the growing Thread footprint. (In your logs, are there lots of these APPARENT DEADLOCKs previously? If you force a JVM Thread dump, do you see lots of com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread instances around, still hung on old, not completing tasks?

Some general comments: If your load is no more than 5-ish simultaneous queries, why such a heavy pool? why not min_size 5-ish, max 10-ish? Your max_statements setting is way too small for the number of Connections you permit in the pool. I'd omit this entirely until you get things working smoothly. Then, to get a bit better performance, you might set the more-straightforward-to-reason-about maxStatementsPerConnection parameter if you want.

Mostly you need to keep your application's footprint (memory? Threads?) well below the resources alotted to it, either by increasing resources, reducing its footprint, or fixing any issue that might exist that cause its resource footprint to increase into limits. I'd start by making the pool smaller, the available memory larger, and configuring the pool to be much smaller.

Answer 3

Authentication issue for me too. I just added the domain part to the database server name and it all worked. Very misleading error.