Java Memory error: unable to create new native thread

Question

I get this error on my UNIX server, when running my java server:

Exception in thread "Thread-0" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at [... where ever I launch a new Thread ...]

It happens everytime I have about 600 threads running.

I have set up this variable on the server:

$> ulimit -s 128

What looks strange to me is the result of this command, which I ran when the bug occured the last time:

$> free -m
              total       used       free     shared    buffers     cached
Mem:          2048        338       1709          0          0          0
-/+ buffers/cache:        338       1709
Swap:            0          0          0

I launch my java server like this:

$> /usr/bin/java -server -Xss128k -Xmx500m -jar /path/to/myJar.jar

My debian version:

$> cat /etc/debian_version
5.0.8

My java version:

$> java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

My question: I have read on Internet that my program should handle something like 5000 threads or so. So what is going on, and how to fix please ?

---

Edit: this is the output of ulimit -a when I open a shell:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 794624
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 794624
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I run the script as a daemon from init.d, and this is what i run:

DAEMON=/usr/bin/java
DAEMON_ARGS="-server -Xss128k -Xmx1024m -jar /path/to/myJar.jar"
ulimit -s 128 && ulimit -n 10240 && start-stop-daemon -b --start --quiet --chuid $USER -m -p $PIDFILE --exec $DAEMON -- $DAEMON_ARGS \
    || return 2

---

Edit2: I have come across this stack overflow question with a java test for threads: how-many-threads-can-a-java-vm-support

    public class DieLikeADog { 
        private static Object s = new Object(); 
        private static int count = 0; 
        public static void main(String[] argv){ 
            for(;;){ 
                new Thread(new Runnable(){ 
                        public void run(){ 
                            synchronized(s){ 
                                count += 1; 
                                System.err.println("New thread #"+count); 
                            } 
                            for(;;){ 
                                try { 
                                    Thread.sleep(100); 
                                } catch (Exception e){ 
                                    System.err.println(e); 
                                } 
                            } 
                        } 
                    }).start(); 
            } 
        } 
    } 

On my server, the program crashes after 613 threads. Now i'm certain this is not normal, and only related to my server configuration. Can anyone help please ?

---

Edit 3: I have come across this article, and many others, explaining that linux can't create 1000 threads, but you guys are telling me that you can do it on your systems. I don't understand.

I have also ran this script on my server: threads_limits.c and the limit is around 620 threads.

My website is now offline and this is the worst thing that could have happened to my project. I don't know how to recompile glibc and this stuff. It's too much work imo.

I guess I should switch to windows server. Because none of the settings proposed on this page did make any change: The limit on my system is between 600 and 620 threads, no matter the program involved.

Answer 1

Just got the following information: This is a limitation imposed by my host provider. This has nothing to do with programming, or linux.

Answer 2

The underlying operating system (Debian Linux in this case) does not allow the process to create any more threads. See here how to raise the maximum amount: Maximum number of threads per process in Linux?

I have read on Internet that my program should handle something like 5000 threads or so.

This depends on the limits set to the OS, amount of running processes etc. With correct settings you can easily reach that many threads. I'm running Ubuntu on my own computer, and I can create around 32000 threads before hitting the limit on a single Java program with all my "normal stuff" running on the background (this was done with a test program that just created threads that went to sleep immediately in an infinite loop). Naturally, that high amount of threads actually doing something would probably screech consumer hardware to a halt pretty fast.

Answer 3

Can you try the same command with a smaller stack size "-Xss64k" and pass on the results ?