Analysing thread dump - lot of blocked threads on sun.misc.Unsafe.park [duplicate]

Question

Working on fixing performance issue in a Java play with akka framework application. Basically consumes and processes messages from a queue. And uses external service APIs heavily while processing each message. I get in to CPU load issue at certain condition, trying to find a root cause. Here is the thread dump of one of host when CPU ~100%.

I see lot of blocked threads with sun.misc.Unsafe.park and do not see any info of application code. Are these blocked ones waiting for IO? Can you give some hints? Thanks

    1032 BLOCKED
threads

    Thread 34542 - threadId:Thread 34542 - state:BLOCKED
    stackTrace:
    - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
    - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame)
    - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
    - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Compiled frame)
    - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
    - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

362 BLOCKED
threads

    Thread 77117 - threadId:Thread 77117 - state:BLOCKED
    stackTrace:
    - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
    - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Compiled frame)
    - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Compiled frame)
    - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
    - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

269 BLOCKED
threads

    Thread 37702 - threadId:Thread 37702 - state:BLOCKED
    stackTrace:
    - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
    - scala.concurrent.forkjoin.ForkJoinPool.scan(scala.concurrent.forkjoin.ForkJoinPool$WorkQueue) @bci=525, line=2075 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool.runWorker(scala.concurrent.forkjoin.ForkJoinPool$WorkQueue) @bci=8, line=1979 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinWorkerThread.run() @bci=14, line=107 (Interpreted frame)

66 BLOCKED
threads

    Thread 33338 - threadId:Thread 33338 - state:BLOCKED
    stackTrace:
    - java.io.ExpiringCache.get(java.lang.String) @bci=0, line=74 (Compiled frame)
    - java.io.UnixFileSystem.canonicalize(java.lang.String) @bci=17, line=152 (Compiled frame)
    - java.io.File.getCanonicalPath() @bci=27, line=618 (Compiled frame)
    - com.azr.rocketman.render.helper.EmailconfigTemplateMethodModel.<init>(java.lang.String) @bci=27, line=50 (Compiled frame)
    - sun.reflect.GeneratedConstructorAccessor84.newInstance(java.lang.Object[]) @bci=28 (Compiled frame)
    - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Compiled frame)
    - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=526 (Compiled frame)
    - freemarker.ext.beans.BeansWrapper.newInstance(java.lang.Class, java.util.List) @bci=144, line=994 (Compiled frame)
    - freemarker.core.NewBI$ConstructorFunction.exec(java.util.List) @bci=32, line=116 (Compiled frame)
    - freemarker.core.MethodCall._eval(freemarker.core.Environment) @bci=52, line=98 (Compiled frame)
      …
    - freemarker.core.Environment.visit(freemarker.core.TemplateElement) @bci=7, line=265 (Compiled frame)
    - freemarker.core.Environment.process() @bci=31, line=243 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.render(java.lang.String, freemarker.template.Configuration, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=165, line=101 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=7, line=60 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=4, line=30 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.renderHtmlBody(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext, com.azr.push.renderer.data.TemplateData) @bci=23, line=146 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext) @bci=61, line=109 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=5, line=63 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=31, line=21 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderAndSendPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=2, line=20 (Compiled frame)
    - com.azr.push.assembler.AbstractDataAssembler.assemble(com.azr.push.pojo.wrapper.UserPayload) @bci=27, line=72 (Compiled frame)
    - com.azr.push.assembler.AssemblerActor.onReceive(java.lang.Object) @bci=39, line=30 (Compiled frame)
    - akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(java.lang.Object, scala.Function1) @bci=7, line=167 (Compiled frame)
    - akka.actor.ActorCell.receiveMessage(java.lang.Object) @bci=25, line=498 (Compiled frame)
    - akka.actor.ActorCell.invoke(akka.dispatch.Envelope) @bci=39, line=456 (Compiled frame)
    - akka.dispatch.Mailbox.processMailbox(int, long) @bci=24, line=237 (Compiled frame)
    - akka.dispatch.Mailbox.run() @bci=20, line=219 (Compiled frame)
    - akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec() @bci=4, line=385 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinTask.doExec() @bci=10, line=260 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(scala.concurrent.forkjoin.ForkJoinTask) @bci=10, line=1339 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool.runWorker(scala.concurrent.forkjoin.ForkJoinPool$WorkQueue) @bci=11, line=1979 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinWorkerThread.run() @bci=14, line=107 (Interpreted frame)

    64 BLOCKED
    threads

    Thread 33194 - threadId:Thread 33194 - state:BLOCKED
    stackTrace:
    - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
    - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame)
    - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
    - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1068 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Compiled frame)
    - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
    - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

    63 BLOCKED
    threads

    Thread 63135 - threadId:Thread 63135 - state:BLOCKED
    stackTrace:
    - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
    - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
    - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
    - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame)
    - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame)
    - org.jboss.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
    - org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=409 (Compiled frame)
    - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=49, line=206 (Compiled frame)
    - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=90 (Interpreted frame)
    - org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
    - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
    - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
    - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
    - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
    - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

    34 blocked threads

    Thread 33336 - threadId:Thread 33336 - state:BLOCKED
    stackTrace:
     …
    - com.azr.push.renderer.processor.FreeMarkerRenderer.render(java.lang.String, freemarker.template.Configuration, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=165, line=101 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=7, line=60 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=4, line=30 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.renderHtmlBody(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext, com.azr.push.renderer.data.TemplateData) @bci=23, line=146 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext) @bci=61, line=109 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=5, line=63 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=31, line=21 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderAndSendPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=2, line=20 (Compiled frame)
    - com.azr.push.assembler.AbstractDataAssembler.assemble(com.azr.push.pojo.wrapper.UserPayload) @bci=27, line=72 (Compiled frame)
    - com.azr.push.assembler.AssemblerActor.onReceive(java.lang.Object) @bci=39, line=30 (Compiled frame)
    - akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(java.lang.Object, scala.Function1) @bci=7, line=167 (Compiled frame)
    - akka.actor.ActorCell.receiveMessage(java.lang.Object) @bci=25, line=498 (Compiled frame)
    - akka.actor.ActorCell.invoke(akka.dispatch.Envelope) @bci=39, line=456 (Compiled frame)
    - akka.dispatch.Mailbox.processMailbox(int, long) @bci=24, line=237 (Compiled frame)
    - akka.dispatch.Mailbox.run() @bci=20, line=219 (Compiled frame)
    - akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec() @bci=4, line=385 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinTask.doExec() @bci=10, line=260 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(scala.concurrent.forkjoin.ForkJoinTask) @bci=10, line=1339 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool.runWorker(scala.concurrent.forkjoin.ForkJoinPool$WorkQueue) @bci=11, line=1979 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinWorkerThread.run() @bci=14, line=107 (Interpreted frame)


    26 blocked threads
    Thread 33327 - threadId:Thread 33327 - state:BLOCKED
    stackTrace:
    - freemarker.core._RegexBuiltins.getPattern(java.lang.String, int) @bci=16, line=122 (Compiled frame)    
      …
    - freemarker.core.Environment.process() @bci=31, line=243 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.render(java.lang.String, freemarker.template.Configuration, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=165, line=101 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=7, line=60 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=4, line=30 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.renderHtmlBody(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext, com.azr.push.renderer.data.TemplateData) @bci=23, line=146 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext) @bci=61, line=109 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=5, line=63 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=31, line=21 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderAndSendPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=2, line=20 (Compiled frame)
    - com.azr.push.assembler.AbstractDataAssembler.assemble(com.azr.push.pojo.wrapper.UserPayload) @bci=27, line=72 (Compiled frame)
    - com.azr.push.assembler.AssemblerActor.onReceive(java.lang.Object) @bci=39, line=30 (Compiled frame)
    - akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(java.lang.Object, scala.Function1) @bci=7, line=167 (Compiled frame)
    - akka.actor.ActorCell.receiveMessage(java.lang.Object) @bci=25, line=498 (Compiled frame)
    - akka.actor.ActorCell.invoke(akka.dispatch.Envelope) @bci=39, line=456 (Compiled frame)
    - akka.dispatch.Mailbox.processMailbox(int, long) @bci=24, line=237 (Compiled frame)
    - akka.dispatch.Mailbox.run() @bci=20, line=219 (Compiled frame)
    - akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec() @bci=4, line=385 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinTask.doExec() @bci=10, line=260 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(scala.concurrent.forkjoin.ForkJoinTask) @bci=10, line=1339 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool.runWorker(scala.concurrent.forkjoin.ForkJoinPool$WorkQueue) @bci=11, line=1979 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinWorkerThread.run() @bci=14, line=107 (Interpreted frame)

    26 blocked thread

    Thread 33333 - threadId:Thread 33333 - state:BLOCKED
    stackTrace:
    - freemarker.core._RegexBuiltins.getPattern(java.lang.String, int) @bci=16, line=122 (Compiled frame)
      ..
    - com.azr.push.renderer.processor.FreeMarkerRenderer.render(java.lang.String, freemarker.template.Configuration, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=165, line=101 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=7, line=60 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=4, line=30 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.renderHtmlBody(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext, com.azr.push.renderer.data.TemplateData) @bci=23, line=146 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext) @bci=61, line=109 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=5, line=63 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=31, line=21 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderAndSendPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=2, line=20 (Compiled frame)
    - com.azr.push.assembler.AbstractDataAssembler.assemble(com.azr.push.pojo.wrapper.UserPayload) @bci=27, line=72 (Compiled frame)
    - com.azr.push.assembler.AssemblerActor.onReceive(java.lang.Object) @bci=39, line=30 (Compiled frame)
    - akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(java.lang.Object, scala.Function1) @bci=7, line=167 (Compiled frame)
    - akka.actor.ActorCell.receiveMessage(java.lang.Object) @bci=25, line=498 (Compiled frame)
    - akka.actor.ActorCell.invoke(akka.dispatch.Envelope) @bci=39, line=456 (Compiled frame)
    - akka.dispatch.Mailbox.processMailbox(int, long) @bci=24, line=237 (Compiled frame)
    - akka.dispatch.Mailbox.run() @bci=20, line=219 (Compiled frame)
    - akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec() @bci=4, line=385 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinTask.doExec() @bci=10, line=260 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(scala.concurrent.forkjoin.ForkJoinTask) @bci=10, line=1339 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool.runWorker(scala.concurrent.forkjoin.ForkJoinPool$WorkQueue) @bci=11, line=1979 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinWorkerThread.run() @bci=14, line=107 (Interpreted frame)

    25 blocked threads

    Thread 33323 - threadId:Thread 33323 - state:BLOCKED
    stackTrace:
    - freemarker.core._RegexBuiltins.getPattern(java.lang.String, int) @bci=16, line=122 (Compiled frame)
      …
    - freemarker.core.Environment.visit(freemarker.core.TemplateElement) @bci=7, line=265 (Compiled frame)
    - freemarker.core.Environment.process() @bci=31, line=243 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.render(java.lang.String, freemarker.template.Configuration, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=165, line=101 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=7, line=60 (Compiled frame)
    - com.azr.push.renderer.processor.FreeMarkerRenderer.renderFromPath(java.lang.String, com.azr.push.renderer.data.TemplateData, com.azr.push.error.MessageError) @bci=4, line=30 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.renderHtmlBody(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext, com.azr.push.renderer.data.TemplateData) @bci=23, line=146 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendEmailContext) @bci=61, line=109 (Compiled frame)
    - com.azr.push.pipeline.rendersend.component.email.RenderEmailComponent.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=5, line=63 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=31, line=21 (Compiled frame)
    - com.azr.push.pipeline.rendersend.RenderAndSendPipeline.process(com.azr.push.pipeline.rendersend.context.RenderAndSendContext) @bci=2, line=20 (Compiled frame)
    - com.azr.push.assembler.AbstractDataAssembler.assemble(com.azr.push.pojo.wrapper.UserPayload) @bci=27, line=72 (Compiled frame)
    - com.azr.push.assembler.AssemblerActor.onReceive(java.lang.Object) @bci=39, line=30 (Compiled frame)
    - akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(java.lang.Object, scala.Function1) @bci=7, line=167 (Compiled frame)
    - akka.actor.ActorCell.receiveMessage(java.lang.Object) @bci=25, line=498 (Compiled frame)
    - akka.actor.ActorCell.invoke(akka.dispatch.Envelope) @bci=39, line=456 (Compiled frame)
    - akka.dispatch.Mailbox.processMailbox(int, long) @bci=24, line=237 (Compiled frame)
    - akka.dispatch.Mailbox.run() @bci=20, line=219 (Compiled frame)
    - akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec() @bci=4, line=385 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinTask.doExec() @bci=10, line=260 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(scala.concurrent.forkjoin.ForkJoinTask) @bci=10, line=1339 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinPool.runWorker(scala.concurrent.forkjoin.ForkJoinPool$WorkQueue) @bci=11, line=1979 (Compiled frame)
    - scala.concurrent.forkjoin.ForkJoinWorkerThread.run() @bci=14, line=107 (Interpreted frame)

Answer 1

sun.misc.Unsafe.park(...) is basically like Thread.wait, but it uses os code, so it is not exposed to us.

You can see in the stack traces that the threads being parked are from thread pools related to blocking queues. Threads that are "parked" coming from pools are simply waiting for a task to be assigned. Also, they really consume 0% CPU so I would doubt this is your issue.

It is possible though that you have a deadlock or concurrency issue making it so your queue is blocking itself forever...

Also, the only thread in there that has anything mentioning I/O is the one with ID 63135.