mvn test java.lang.OutOfMemoryError: unable to create new native thread

Question

When I run mvn test I receive the below exception. I have tried both raising and lowering my Xmx and Xss JVM settings and bumping all limits under ulimit. There are about 1300 tests and the last 200 always fail with this exception. Running those tests by themselves allows them to pass. The same tests pass on my ubuntu box. I receive this exception when running the tests on my mac. I am pretty sure its an environment issue, but I have tweaked every setting that I am aware of with absolutely no luck.

I am using mvn 2.1 and Java 6. My test framework is junit 4.8.

java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:658)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
        at com.google.appengine.tools.development.ApiProxyLocalImpl$PrivilegedApiAction.run(ApiProxyLocalImpl.java:197)
        at com.google.appengine.tools.development.ApiProxyLocalImpl$PrivilegedApiAction.run(ApiProxyLocalImpl.java:184)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.google.appengine.tools.development.ApiProxyLocalImpl.doAsyncCall(ApiProxyLocalImpl.java:172)
        at com.google.appengine.tools.development.ApiProxyLocalImpl.makeAsyncCall(ApiProxyLocalImpl.java:138)

Answer 1

I ran into this problem using the Maven Surefire plugin (v2.12). I had to configure surefire as follows:

<configuration>
  <forkMode>always</forkMode>
  ... other surefire config ...
</configuration>

If I omitted the "forkMode" element in my config, I would get the "unable to create new native thread" error because the java Surefire process would create thousands of threads otherwise, exceeding my OS limit (Mac OSX -- you can see this in the Activity Monitor).

As best I can tell, all the new threads are getting created because the default "forkMode" is "once" in Surefire, and whatever new threads are getting created don't die off until the "one" surefire process terminates.

One final note: adjusting my JVM memory settings seemed to have no effect (good or bad). Using default values worked normally, as did the following:

<argLine>-Xss512k -Xms512m -Xmx4096m -XX:MaxPermSize=2048m</argLine>

Answer 2

Related to sappenin's answer, you should use the forkCount and reuseForks configuration parameters (instead of deprecated forkMode) for newer versions of the surefire plugin. The resulting plugin configuration might look like the shown code.

<build>
    <plugins>
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.16</version>
            <configuration>
                <forkCount>1</forkCount>
                <reuseForks>false</reuseForks>
                <argLine>-Xms256m -Xmx1024m</argLine> 
            </configuration>
        </plugin>
    </plugins>
</build>

citation: http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html