How to detect scala executioncontext exhaustion?

Question

I'm having problems with my Playframework application not being responsive from time to time and I would like to detect this at runtime + log information on what is currently running on the exhausted execution context.

What would the best strategy for implementing this be? I thought about posting small runnables to the execution contexts and if they don't get executed in time I would log a warning. This max wait time should of course be configurable. Eg the main web execution context should never be blocked for more than 1 sec but a background db execution context might allow 30 sec of blocking.

Somebody must have done this before?

related info: http://www.playframework.com/documentation/2.2.x/ThreadPools

Answer 1

That is pretty hard question to answer.

Configuration of dispatchers really depends on the type of work your actors are doing.

Probably you should look at the actors that spawn futures to do their work. It may be a good idea to predefine execution context in the configuration file and use them like this:

implicit val ec : ExecutionContext = context.system.dispatchers.lookup("someDispatcher")

those actors are likely to cause the starvation effect.

The effect you want to achieve is to make sure messages are processed quickly and long running task does not affect them so separation is the key thing here.

A good tool for monitoring your application is the Typesafe Console. You can look at the dispatcher there and see that the latency in handling messages increase.

Another concern is to identify actors that make a high risk work like network I/O. If something happens to the thread making in not available again in the pool it will create problems.

It is very likely that without experimenting with the threadpool size you won't know what is the best setting for you.

Most of those advises I wrote I know from the book Effective Akka by Jamie Allen and they seem to be working quite well for me. There is a section in this book Fixing Starvation. You may want to look at it closer.