Strategy for debugging surefire "The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?"

Question

I am working on a rather complex java project with many dependencies and many unit tests.

I am using java 1.6.0_65 on mac (mavericks) with maven 3.0.5 with maven-surefire-plugin:2.16 running in several forks. My problem is that running this setup with several forks causes a fork to exit with:

"The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?"

running this with only one fork does not produce the problem (and everything passes)

There is some information out there about this problem including this StackOverflow question and this surefire bug (which seems to be solved by now)

I am aware that the "Answer" to this situation is to find what in my code calls System.exit() - Nothing i could find.

Or what causes my JVM to crash - there are no hs_pid crash reports.

My Question is what what kind of strategy can i use to find a locate this cause? To clarify, I am not interested in the answer noted above, but a way to find where it is originating from. (Or even better a different answer altogether to what might be causing this )

---

my Surefire configuration is: (but i did try other combinations)

<parallel>classes</parallel>
<threadCount>1</threadCount>
<forkCount>1C</forkCount>
<reuseForks>false</reuseForks>
<useSystemClassLoader>false</useSystemClassLoader>
<useManifestOnlyJar>true</useManifestOnlyJar>
<useFile>true</useFile>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<runOrder>alphabetical</runOrder>

---

Update #1 adding the relevant out put after running the maven goal with --debug (-X)

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project event-logger: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
[ERROR] Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project event-logger: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.apache.maven.plugin.MojoFailureException: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.SurefirePlugin.assertNoException(SurefirePlugin.java:198)
    at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:188)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:852)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:720)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    ... 25 more
Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:316)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:169)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:958)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:822)
    ... 28 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:300)
    ... 31 more
Caused by: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:485)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:352)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$300(ForkStarter.java:85)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:288)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:283)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Answer 1

Steps:

(1) Run mvn with the -e and -X options to get more debug information.

(2) Look for "Error" in the output. In my case, when I ran the mvn command, part of the output included:

[ERROR] Command wascmd.exe /X /C "C:\dev\dev-tools\.....

(3) Execute the problematic command directly in the command shell.

In my case, executing

cmd.exe /X /C "C:\dev\dev-tools\....

from the command line resulted in an OutOfMemoryError.

Answer 2

In my case the configuration forkedProcessExitTimeoutInSeconds for the maven-surefire-plugin helps. The default value are since maven-surefire-plugin:2.20.1 30 seconds. My project gots very time consuming test and so the forked JVM chrashes. Configure the plugin in the pom with the following property solves the problem.

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
                <forkedProcessExitTimeoutInSeconds>120</forkedProcessExitTimeoutInSeconds>
        </configuration>
</plugin>

Answer 3

If anyone is including a custom argLine argument, you must reconsider because it is likely the source of your issues with the memory allocation.

For Example (I used to have):

<argLine>XX:MaxPermSize=4096m ${argLine}</argLine>

Now I use hard specified values:

<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>

For whatever reason, Applications that integrate with Surefire such as Jacoco, dont request enough memory to coexist with the testing that happens at build time.

(more information can also be found at this s.o. question (1))

(1)-maven jacoco: not generating code coverage report

Answer 4

Writing here the strategy that i used to help others who are stuck with this problem.

It is possible to leverage the SecurityManager to throw an exception when System.exit() is called. you can then examine the stack trace to see exactly who called exit(). This is especially useful if the call to exit was made from code hidden inside one of the jars you are depending on and not from your own code.

private static void forbidSystemExitCall() {
    final SecurityManager securityManager = new SecurityManager() {
        public void checkPermission( Permission permission ) {
            if( permission.getName().startsWith("exitVM") ) {
                throw new RuntimeException("Something called exit ") ;
            }
        }
    } ;
    System.setSecurityManager( securityManager ) ;
}  

Answer 5

Following Maven Surefire documentation, you can execute the forked VM in debug mode, if it always fails. You can then debug your code until it exit.