Jenkins hangs between build and post-build

Question

After updating Jenkins to version 2.156 (from version 1.6), some of our build jobs get stuck after completing and before moving to post-build action. Job itself is finished within 5 minutes (same as before), then it hangs for 5-10 minutes before moving on.

I managed to narrow it down to this:

"Executor #10 for master : executing 03_masa #4390" Id=34464 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at hudson.util.ProcessTree$WindowsOSProcess.killSoftly(ProcessTree.java:560)
    at hudson.util.ProcessTree$WindowsOSProcess.killRecursively(ProcessTree.java:520)
    at hudson.util.ProcessTree$Windows.killAll(ProcessTree.java:666)
    at hudson.Launcher$LocalLauncher.kill(Launcher.java:955)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510)
    at hudson.model.Run.execute(Run.java:1810)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)

Referenced code can be found here (present since version 2.141).

threadDump #1, threadDump #2

Can we do something about it?

Answer 1

2.141 introduced a 2min wait on process termination (it would seem that multiplies with as many processes as were created during your build)

https://github.com/jenkinsci/jenkins/commit/d8eac92ee9a1c19bf145763589f1c152607bf3ed

unsure why killSoftly does not work but you can configure the timeout

In your jenkins.xml you can add this to your /service/arguments element (before the -jar) like so:

-DSoftKillWaitSeconds=0

After doing so and restarting jenkins you should be able to find your SoftKillWaitSeconds setting under /systemInfo

and your build time should be back to normal