Akka warning about "Too many HashedWheelTimer instances"

Question

I'm getting the following warning when my Akka 1.2-based application tries to work with more than 500 or so grid nodes:

Jan 05, 2012 1:36:43 PM org.jboss.netty.util.internal.SharedResourceMisuseDetector
WARNING: You are creating too many HashedWheelTimer instances.  
HashedWheelTimer is a shared resource that must be reused across the application, so that only a few instances are created.

I'm going to dig more deeply into the Akka code to find out where the HashedWheelTimer is being used and what I should do to avoid too many of those being created. While I'm doing that, I thought I'd post the warning here in case one of the experts may be able to give me a pointer or two that may narrow and speed up my search, please?

Using Akka 1.3-RC6, I'm getting the following exception:

Jan 12, 2012 1:24:38 PM org.jboss.netty.util.HashedWheelTimer
WARNING: An exception was thrown by TimerTask.
org.jboss.netty.channel.ChannelException: Failed to open a socket.
    at org.jboss.netty.channel.socket.nio.NioClientSocketChannel.newSocket(NioClientSocketChannel.java:49)
    at org.jboss.netty.channel.socket.nio.NioClientSocketChannel.<init>(NioClientSocketChannel.java:83)
    at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:139)
    at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:86)
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:218)
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188)
    at akka.remote.netty.ActiveRemoteClient.connect(NettyRemoteSupport.scala:470)
    at akka.remote.netty.ActiveRemoteClientHandler$$anonfun$channelClosed$1$$anon$4.run(NettyRemoteSupport.scala:599)
    at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:519)
    at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:440)
    at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:379)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Too many open files
    at sun.nio.ch.Net.socket0(Native Method)
    at sun.nio.ch.Net.socket(Net.java:323)
    at sun.nio.ch.Net.socket(Net.java:316)
    at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:101)
    at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
    at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
    at org.jboss.netty.channel.socket.nio.NioClientSocketChannel.newSocket(NioClientSocketChannel.java:47)
    ... 11 more

Answer 1

Has been fixed in Akka 2.0-M1, I've now backported the fix to 1.3 and it will be in the next RC.