java.net.SocketException: Cannot allocate memory (not Mac)

Question

I have a java app deployed on tomcat 7 running on ubuntu 10.04. There's been an issue during opening a server socket which I couldn't reproduce so far:

java.net.SocketException: Cannot allocate memory
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(Unknown Source)
at java.net.ServerSocket.bind(Unknown Source)
at org.subethamail.smtp.server.SMTPServer.createServerSocket(SMTPServer.java:338)
at org.subethamail.smtp.server.SMTPServer.start(SMTPServer.java:291)

All I've been able to find out is that this happens on some specific version of MacOS which is not relevant for me, and also on OpenJDK, which is not relevant either (I'm using Oracle JRE 1.7.0_17). Another possible reason is a virtualization environment, but in my case this happens on a hardware box.

So, the question is, has anyone ever faced the same problem and what could be a possible solution.

Update There's been also this thing: tomcat consumed almost all of the heap, approximately 700mb, it's been caused by a memory leak in my code. But as far as I understand, the exception tells about a socket buffer on system level, so it doesn't seem to be related to java heap. However, this is the only explanation I've got so far and it's very illusive in my opinion.

Update 2 Eventually we've been able to reproduce the issue several times, so this was not about memory leaks. I was considering authbind as a possible source of the problem when I faced it for the first time, but unfortunately I haven't paid much attention to it. When I got another hardware box affected by the problem, I tried to bind non-priveleged port and succeeded, while attempts to bind priveleged ports lead to exceptions. So, eventually I've replaced authbind with iptables.

Basically, fady taher's answer points to authbind, but Danny Thomas's answer provides very interresting information about connections between forking and "Cannot allocate memory", actually we also use process builder to run bash scripts, so there is a great chance that the problem could be caused by it.

Answer 1

Sounds like you have insufficient physical memory or swap - on the systems affected, check memory and swap.

Does your application happen to execute external commands - fork/exec could be contributing. You might consider allowing memory overcommit, if that's the case:

http://bryanmarty.com/blog/2012/01/14/forking-jvm/