How to set `killSoftly` for a specific Jenkins job?

Question

My Jenkins build hangs between build and post-build steps.

The console output shows there is a 6-minute wait (but I've seen waits of up to one hour):

10:53:26 BUILD FAILED in 1m 7s
10:53:26 4 actionable tasks: 4 executed
10:53:26 Build step 'Invoke Gradle script' changed build result to FAILURE
10:53:26 Build step 'Invoke Gradle script' marked build as failure
11:09:29 [CucumberReport] Using Cucumber Reports version 4.9.0

I found this and this questions that have similar issues, and they say the solution is setting -DSoftKillWaitSeconds=0 in jenkins.xml.

However, I need a way to set the option for particular jobs only, without messing with global Jenkins settings (I wouldn't want to mess with other projects).

EDIT:

When I manually abort the job, before the [CucumberReport] step, Cucumber reports are still generated.

I also checked Abort the build if it's stuck checkbox in Build Environment options, with Time-out strategy set to No Activity (Timeout seconds = 2).

When I build the project with these settings, the build will fail with "Aborted after 0 seconds" shown in Build History, as before, but the console output will be the same. (Nothing changes, Cucumber Reports will be generated but after a certain timeout).

Answer 1

It is not possible to select a job-specific value for SoftKillWaitSeconds (the value is derived from the Jenkins core at a point where the job name is not known).

My recommendation is to fix the abort handling in your job itself, so it will not depend on a "soft kill timeout". If you're running on a Unix-ish system, you can ensure this by running your job in a new process group (set -m in bash) and (for example) setting up a proper exit trap.

Answer 2

We are using the Build-timeout plugin to kill stuck jobs with timeout strategy set to No Activity or Absolute. For me, this is a good approach when you are using freestyle projects. The reason why your build is "Aborted after 0 seconds" is that most likely there are unfinished child processes. From documentation:

Because Java only allows threads to be interrupted at a set of fixed locations, depending on how a build hangs, the abort operation might not take effect. For example,

• if Jenkins is waiting for child processes to complete, it can abort right away.
• if Jenkins is stuck in an infinite loop, it can never be aborted.
• if Jenkins is doing a network or file I/O within the Java VM (such as lengthy file copy or SVN update), it cannot be aborted.

You could try the absolute timeout strategy. You can define a global variable, so that you do not repeat the timeout value in jobs:

1. Go to "Manage Jenkins" > "Configure System".
2. Check "Environment variables" in "Global properties".
3. Add an environment variable name="GLOBAL_TIMEOUT_MINUTES" value="20".
4. Go to a configuration page of a project.
5. Check "Abort the build if it's stuck" in "Build Environment".
6. Select "Absolute" for "Time-out strategy". Of course, also applicable to other strategies.
7. Set "${GLOBAL_TIMEOUT_MINUTES}" for "Timeout".
8. Set timeout action "Abort the build".

If this is not working, you could try to look in the logs https://your-jenkins-server/log or in a thread dump.
The hanging may be caused by new/old version of a plugin. Try to find what are the unfinished child processes. Try to disable post-build actions one by one to find the one that may be the cause of the issue. You can see https://superuser.com/questions/1401879/debugging-what-happens-when-a-jenkins-build-unexpectedly-pauses-or-hangs