How to profile Akka applications?

Question

I have a small Akka application that passes many messages between its actors and each actor does some calculations on the data it receives. What I want is to profile this application in order to see which parts of the code take up most time and so on.

I tried VisualVM but I cannot really understand what's going on. I added a picture of the profiler output.

My questions are

• What for example is this first line and why does it take up so much time? (scala.concurrent.forkjoin.ForkJoinPool.scan())
• Can Akka applications because of their asynchronous behaviour be profiled well at all?
• Can I see for instance how long one specific actor(-type) works for one specific message(-type) it receives?
• Are there other best-practices for profiling Akka applications?

Answer 1

• There are packages not profiled by default and it is their time that is accounted in the profile of scala.concurrent.forkjoin.ForkJoinPool.scan(). If all the hidden packages are allowed to be sampled, the true CPU time consumers will be revealed. For example, the following before/after illustrative profiles uncover that threads are put to sleep most of the time by sun.misc.Unsafe.park waiting to be unparked.

• Akka applications can be profiled quite well with proper instrumentation and call tracing. Google's prominent Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper contains detailed explanation of the technique. Twitter created Zipkin based on that. It is open sourced and has an extension for distributed tracing of Akka. Follow its wiki for a good explanation of how to set up a system that allows to

  • trace call hierarchies inside an actor system;
  • debug request processing pipelines (you can log to traces, annotate them with custom key-value pairs);
  • see dependencies between derived requests and their contribution to resulting response time;
  • find and analyse slowest requests in your system.

  There is also a new kid on the block, Kamon. It is a reactive-friendly toolkit for monitoring applications that run on top of the JVM, which is specially enthusiastic to applications built with the Typesafe Reactive Platform. That definitely means yes for Akka and the integration comes in the form of the kamon-akka and kamon-akka-remote modules that bring bytecode instrumentation to gather metrics and perform automatic trace context propagation on your behalf. Explore the documentation starting from Akka Integration Overview to understand what it can and how to achieve that.