On our server, we started to have problems with OutOfMemoryError
. We analyzed the heap dumps using Eclipse Memory Analysis, and found, that many objects were held to do finalization (about 2/3 of the heap):
We found, that it could be some finalize() method blocking. I found several bug reports of this problem (here or here), and it always manifested itself in the Finalizer thread stack, that it was blocked somewhere. But in our case, this thread was WAITING:
"Finalizer" daemon prio=10 tid=0x43e1e000 nid=0x3ff in Object.wait() [0x43dfe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x4fe053e8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x4fe053e8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
EDIT:
We then tried to add -XX:+UseConcMarkSweepGC
, but with no success, only the frequency of OutOfMemoryError
s diminished, so we first thought it helped.
Finally, we suspected the JVM bug and upgraded from OpenJDK 1.6.0_30 to Oracle JDK 1.7.0_51, and the problem disappeared (at least it seems so, during last 4 hours the used heap does not grow). We do not remember any change in finalize method, nor did we upgrade any library, there were only minor developments during that time. The problem does not reproduce on our test server, with same configuration except that it is 64bit JVM while the production server is 32bit.
The question is: what could be the cause for Objects not being finalized and Finalizer
thread waiting for next object? Did we analyze the heap dump correctly?
Thanks for all answers.
We think it was related to the OpenJDK version 1.6.0_30. After upgrading to Oracle JDK 1.7.0_51, the problem disappeared. And it probably appeared after automatic update of openJDK, but we cannot confirm this either. We could not find a relevant bug report.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With