Best way on how to solve/debug JVM crash (SIGSEGV)

Question

I'm really lost and I don't know how to face and solve my problem. I have a piece of simple Java Code, which leads to a JVM crash:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001057ce9d4, pid=10727, tid=18947
#
# JRE version: Java(TM) SE Runtime Environment (8.0_73-b02) (build 1.8.0_73-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.73-b02 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.dylib+0x3ce9d4]  PhaseIdealLoop::idom_no_update(Node*) const+0x12
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007feeef003800):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=18947, stack(0x0000700000ec4000,0x0000700000fc4000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000008

I have no idea on how to solve the problem. The program is pretty simple, it receives a message through Kafka and triggers tasks based on the message received. If I add two different tasks, the program crashes after 900 - 1,500 messages. All of the messages are the same and the program does not use any JNI stuff (the used 3rd party libraries don't use any JNI as well, as far as I'm informed).

I never faced this problem, but I'd love/need to find a way on how to figure out what the problem is. I already used other versions of the JVM (Java 8.0_66, 8.0_73-b02, and 8.0_74-b02). So what can I do? Thank you so much!

EDIT (1): Sometimes I also get the following error/info:

...
# JRE version: Java(TM) SE Runtime Environment (8.0_73-b02) (build 1.8.0_73-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.73-b02 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.dylib+0x3ce9d4]
...

EDIT (2): I updated my Java version to 8.0_74. The error is still there :(.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001073cdef8, pid=11227, tid=19715
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.dylib+0x3cdef8]  PhaseIdealLoop::idom_no_update(Node*) const+0x12
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f89e481c800):  JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=19715, stack(0x000070000104a000,0x000070000114a000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000008

EDIT (3): Core Dump

So finally I created a core dump and loaded it into Java VisualVM (I could not use the solution presented by DROY because calling jmap lead to another error: "Error attaching to core file: Can't attach to the core file"). The threaddump created with VisualVM results in:

Thread 30239 "Keep-Alive-Timer": (state = BLOCKED)
    at java.lang.Thread.sleep(Native Method)
    at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:172)
    at java.lang.Thread.run(Thread.java:745)

Thread 29699 "threadDeathWatcher-4-1": (state = BLOCKED)
    at java.lang.Thread.sleep(Native Method)
    at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:745)

Thread 26635 "nioEventLoopGroup-3-1": (state = IN_NATIVE)
    at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
    at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
    at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000006c049ec98> (a io.netty.channel.nio.SelectedSelectionKeySet)
    - locked <0x00000006c049ec88> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006c049ecb8> (a sun.nio.ch.KQueueSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:745)

Thread 29187 "pool-3-thread-1": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:63)
    at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
    at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
    at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
    at com.sosse.common.messaging.DefaultHandler.doRun(DefaultHandler.java:22)
    at com.sosse.common.concurrency.DefaultRunnable.run(DefaultRunnable.java:11)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Thread 28675 "pool-4-thread-1": (state = BLOCKED)
    at java.lang.Thread.sleep(Native Method)
    at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:461)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:360)
    at java.lang.Thread.run(Thread.java:745)

Thread 28163 "ConsumerFetcherThread-analytics-group_Philipp.local-1458441725398-581eabc3-0-0": (state = IN_NATIVE)
    at sun.nio.ch.Net.poll(Native Method)
    at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
    - locked <0x00000006c056d538> (a java.lang.Object)
    at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204)
    - locked <0x00000006c056d5b8> (a java.lang.Object)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    - locked <0x00000006c056d5f8> (a sun.nio.ch.SocketAdaptor$SocketInputStream)
    at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
    - locked <0x00000006c056d618> (a java.lang.Object)
    at kafka.utils.Utils$.read(Utils.scala:380)
    at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
    at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
    at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
    at kafka.network.BlockingChannel.receive(BlockingChannel.scala:111)
    at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:71)
    at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
    - locked <0x00000006c056d6e0> (a java.lang.Object)
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:112)
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:112)
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:112)
    at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:111)
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:111)
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:111)
    at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
    at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:110)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:94)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

Thread 27651 "analytics-group_Philipp.local-1458441725398-581eabc3-leader-finder-thread": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:61)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

Thread 27139 "analytics-group_Philipp.local-1458441725398-581eabc3_watcher_executor": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
    at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:544)

Thread 26115 "kafka-consumer-scheduler-0": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Thread 25603 "main-EventThread": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

Thread 25091 "main-SendThread(localhost:2181)": (state = IN_NATIVE)
    at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
    at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
    at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000006c0022c50> (a sun.nio.ch.Util$2)
    - locked <0x00000006c0022c60> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006c0022c00> (a sun.nio.ch.KQueueSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

Thread 24579 "ZkClient-EventThread-16-localhost:2181": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:67)

Thread 24067 "metrics-meter-tick-thread-2": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Thread 23555 "metrics-meter-tick-thread-1": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Thread 23303 "pool-1-thread-1": (state = BLOCKED)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

VM Thread 20995 "Service Thread": (state = BLOCKED)

VM Thread 20483 "C1 CompilerThread3": (state = BLOCKED)

VM Thread 19971 "C2 CompilerThread2": (state = IN_NATIVE)

VM Thread 19459 "C2 CompilerThread1": (state = IN_NATIVE)

VM Thread 18947 "C2 CompilerThread0": (state = IN_NATIVE)

Thread 15887 "Signal Dispatcher": (state = BLOCKED)

Thread 14339 "Finalizer": (state = BLOCKED)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000006c005fa88> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    - locked <0x00000006c005fa88> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

Thread 13827 "Reference Handler": (state = BLOCKED)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000006c0029358> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    - locked <0x00000006c0029358> (a java.lang.ref.Reference$Lock)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Thread 2823 "main": (state = BLOCKED)
    at java.lang.Thread.sleep(Native Method)
    at com.sosse.analytics.ActivityProducer.sleep(ActivityProducer.java:110)
    at com.sosse.analytics.DemoMain$1.startApplication(DemoMain.java:37)
    at com.sosse.common.application.DefaultApplication.start(DefaultApplication.java:166)
    - locked <0x00000006c00e6080> (a java.lang.Thread)
    at com.sosse.common.application.DefaultApplication.start(DefaultApplication.java:118)
    at com.sosse.analytics.DemoMain.main(DemoMain.java:48)

EDIT (4): Source Code Deserializer

public static Object[] deserialize(final JsonElement jsonElement, final JsonDeserializationContext context, final BiFunction<Class<?>, JsonElement, Serializable[]> timeSeriesDeserializer) {
    final JsonObject jsonObject = jsonElement.getAsJsonObject();

    // get the important classes
    final Class<?> bucketContent = resolveClass("bucketContent", jsonObject, context);

    // configuration
    final TimeUnit timeUnit = context.deserialize(jsonObject.get("timeUnit"), TimeUnit.class);
    final int bucketSize = context.deserialize(jsonObject.get("bucketSize"), int.class);
    final boolean fillNumberWithZero = context.deserialize(jsonObject.get("fillNumberWithZero"), boolean.class);

    // the values
    final Long now = context.deserialize(jsonObject.get("now"), Long.class);
    final Serializable[] timeSeries = timeSeriesDeserializer.apply(bucketContent, jsonObject.get("timeSeries"));

    @SuppressWarnings("unchecked")
    final BucketTimeSeriesConfig config = new BucketTimeSeriesConfig(bucketContent, timeUnit, timeSeries.length, bucketSize, fillNumberWithZero);
    return new Object[]{config, timeSeries, now};
}

Answer 1

The crash report tells the error has happened in JIT compiler thread:

Current thread (0x00007f89e481c800):  JavaThread "C2 CompilerThread1"

Take the following steps do diagnose compiler problems:

1. Try the most recent JVM build available in JDK 9 EA: https://jdk9.java.net/download/

   If the problem disappears, you can either stay with this version or try to locate the exact commit that solves the issue and then backport it to JDK 8. How to backport fixes and how to build HotSpot yourself - it's a separate topic, but I can tell if you're interested.

2. If the problem persists, try to find a problematic method and exclude it from compilation.

   Current CompileTask: C2: 114667 5303 4 com.sosse.time.timeseries.gson.TypeConverterHelper::deserialize (157 bytes)
   

   Looks like in your case it fails compiling TypeConverterHelper.deserialize(). Add the following JVM option to exclude this particular method:

   -XX:CompileCommand=exclude,com.sosse.time.timeseries.gson.TypeConverterHelper::deserialize
   

3. If it does not help, try to exclude more methods by providing multiple -XX:CompileCommand. To find candidates to exclude use -XX:+PrintCompilation and look at the bottom of the printed list. You can also exclude the whole classes and packages from compilation, e.g.

   -XX:CompileCommand=exclude,com.sosse.time.timeseries.gson.*::*
   

4. Try to disable certain compiler optimizations one by one. Some options to try are:

   -XX:-DoEscapeAnalysis
   -XX:LoopUnrollLimit=0
   -XX:-PartialPeelLoop
   -XX:-UseLoopPredicate
   -XX:-LoopUnswitching
   -XX:-ReassociateInvariants
   -XX:MaxInlineLevel=1
   -XX:-IncrementalInline
   -XX:-RangeCheckElimination
   -XX:-EliminateAllocations
   -XX:-UseTypeProfile
   -XX:AliasLevel=0
   

5. Whether the problematic method/optimization is found or not, run JVM again with

   -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation
   

   This will create hotspot_pid1234.log file in the current directory with detailed compilation log.

6. Submit the bug report at bugreport.java.com. Select

   Product/Category: HotSpot Virtual Machine (errors)
   Subcategory:      J2SE Server Compiler
   

   Make sure to include full hs_err_pid.log and hotspot_pid.log from step 5. It would be very helpful if you could provide a reduced self-contained example that demonstrates the problem.

   For a faster reaction you may also post a message to hotspot-compiler-dev mailing list.