start daemon on remote server via Jenkins SSH shell script exits mysteriously

Question

I have a build job on jenkins that is building my project and after it is done, it opens an ssh shell script on a remote server and transfers files and then stop and starts a daemon.

When I stop and start the daemon from the command line on a RHEL server, it executes just fine. When the job executes in jenkins, there are no errors.

The daemon stops fine and it starts fine. But shortly after starting, the daemon dies suddenly.

sudo service daemonName stop
# transfer files.
sudo service daemonName start

I'm sure that the problem isn't pathing

Does anyone know what could be special about the way Jenkins is executing the ssh shell script that would cause the daemon start to not fully complete?

Answer 1

The problem: When executing a build through jenkins, the command to start the daemon process was clearly successfully executing, yet after the build job was done, the daemon would suddenly quit.

The solution: I thought for this whole time that it was jenkins killing the daemon. So I tried many different incarnations and permutations of disabling the ProcessTree module that goes through and cleans up zombie child processes. I tried fooling it by resetting the BUILD_ID environment variable. Nothing worked.

Thanks to this thread I found out that that solution only works for child processes executed on the BUILD machine. I.E. not applicable to my problem.

More searching led me here: Run a persistent process via ssh

The solution? Nohup.

So now the build successfully restarts the daemon by executing the following: sudo nohup service daemonname start

Answer 2

Jenkins watches for processes spawned by the job and kill them to avoid zombie processes. See https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller

The workaround is to override the BUILD_ID environment variable:

BUILD_ID=dontKillMe