One day ago, after a few months of normal working, our java app starts to crash occasionally with the following error:
# # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (safepoint.cpp:247), pid=2075, tid=140042095163136 # guarantee(PageArmed == 0) failed: invariant # # JRE version: 6.0_23-b05 # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode linux-amd64 compressed oops) # An error report file with more information is saved as: # /var/chat/jSocketer/build/hs_err_pid2075.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp #
I looked in hs_err_pid2075.log and saw that there was an active thread, that processed a network communication. However there wasn't any application or environment changes done in the last few months. Also there wasn't any load growth. What can I do to understand, what is the reason of crash? Are there any common steps to investigate a jvm crash?
UPD http://www.wuala.com/ubear/public
The JVM crash log location is specified in run. bat on Windows, or run.sh on Linux, and is enabled by default.
There are various possible reasons for a crash. For example, a crash can occur due to a bug in the Java HotSpot VM, in a system library, in a Java SE library or an API, in application native code, or even in the operating system (OS). External factors, such as resource exhaustion in the OS can also cause a crash.
The crash could of course be caused by something else, but analysis of the library and any core file or crash dump is a good starting place.
Reduce the Java heap size. The Java heap is only a certain part of the total memory used by the JVM. If the Java heap is significantly larger, JVM can run out of virtual memory while compiling methods or when native libraries are loaded. Try lowering the maximum heap size to avoid this error.
The crash is in the JVM, not in external native code. However, the operation it crashed on has been initiated by and external DLL.
This line in the hs_err_pid file explains the operation that crashed:
VM_Operation (0x00007f5e16e35450): GetAllStackTraces, mode: safepoint, requested by thread 0x0000000040796000
Now, thread 0x0000000040796000 is
0x0000000040796000 JavaThread "YJPAgent-Telemetry" daemon [_thread_blocked, id=2115, stack(0x00007f5e16d36000,0x00007f5e16e37000)]
which is a thread created by Yourkit. "GetAllStackTraces" is something that a profiler needs to call in order to do sampling. If you remove the profiler, the crash will not happen.
With this information It's not possible to say what causes the crash, but you can try the following: Remove all -XX VM parameters, -verbose:gc and the debugging VM parameters. They might interfere with the profiling interface of the JVM.
Update
Code that calls java.lang.Thread#getAllStackTraces()
or java.lang.Thread#getStackTrace()
may trigger the same crash
The two times I've witnessed recurring JVM crashes were both due to hardware failure, namely RAM. Running a memtest utility is the first thing I'd try.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With