"java.lang.OutOfMemoryError : unable to create new native Thread"

Question

This is not a memory problem even though the exception name highly suggests so, but an operating system resource problem. You are running out of native threads, i.e. how many threads the operating system will allow your JVM to use.

This is an uncommon problem, because you rarely need that many. Do you have a lot of unconditional thread spawning where the threads should but doesn't finish?

You might consider rewriting into using Callable/Runnables under the control of an Executor if at all possible. There are plenty of standard executors with various behavior which your code can easily control.

(There are many reasons why the number of threads is limited, but they vary from operating system to operating system)

I encountered same issue during the load test, the reason is because of JVM is unable to create a new Java thread further. Below is the JVM source code

if (native_thread->osthread() == NULL) {    
// No one should hold a reference to the 'native_thread'.    
    delete native_thread;   
if (JvmtiExport::should_post_resource_exhausted()) {      
    JvmtiExport::post_resource_exhausted(        
        JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | 
        JVMTI_RESOURCE_EXHAUSTED_THREADS, 
        "unable to create new native thread");    
    } THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(), "unable to create new native thread");  
} Thread::start(native_thread);`

Root cause : JVM throws this exception when JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR (resources exhausted (means memory exhausted) ) or JVMTI_RESOURCE_EXHAUSTED_THREADS (Threads exhausted).

In my case Jboss is creating too many threads , to serve the request, but all the threads are blocked . Because of this, JVM is exhausted with threads as well with memory (each thread holds memory , which is not released , because each thread is blocked).

Analyzed the java thread dumps observed nearly 61K threads are blocked by one of our method, which is causing this issue . Below is the portion of Thread dump

"SimpleAsyncTaskExecutor-16562" #38070 prio=5 os_prio=0 tid=0x00007f9985440000 nid=0x2ca6 waiting for monitor entry [0x00007f9d58c2d000]
   java.lang.Thread.State: BLOCKED (on object monitor)

If jvm is started via systemd, there might be a maxTasks per process limit (tasks actually mean threads) in some linux OS.

You can check this by running "service status" and check if there is a maxTasks limit. If there is, you can remove it by editing /etc/systemd/system.conf, adding a config: DefaultTasksMax=infinity

It's likely that your OS does not allow the number of threads you're trying to create, or you're hitting some limit in the JVM. Especially if it's such a round number as 32k, a limit of one kind or another is a very likely culprit.

Are you sure you truly need 32k threads? Most modern languages have some kind of support for pools of reusable threads - I'm sure Java has something in place too (like ExecutorService, as user Jesper mentioned). Perhaps you could request threads from such a pool, instead of manually creating new ones.

I would recommend to also look at the Thread Stack Size and see if you get more threads created. The default Thread Stack Size for JRockit 1.5/1.6 is 1 MB for 64-bit VM on Linux OS. 32K threads will require a significant amount of physical and virtual memory to honor this requirement.

Try to reduce the Stack Size to 512 KB as a starting point and see if it helps creating more threads for your application. I also recommend to explore horizontal scaling e.g. splitting your application processing across more physical or virtual machines.

When using a 64-bit VM, the true limit will depend on the OS physical and virtual memory availability and OS tuning parameters such as ulimitc. I also recommend the following article as a reference:

OutOfMemoryError: unable to create new native thread – Problem Demystified