How do I run, test, and terminate an HTTP server using Jenkins?

Question

I'm working on a team that is building a RESTful HTTP service. We're having trouble with setting up a Jenkins CI job which will build the service, run it in the background, execute some tests, and then terminate the servers.

Specifics

• The server is built in Node.js using the hapi framework and has some unit tests written in mocha.
• The tests are written in Java using Maven. (Why not node.js-based tests? Because our testing dept. has invested time in creating a Java-based REST-testing framework.)
• The build should fail if the node-based unit tests fail or if the java tests fail.
• Our Jenkins box is run by a support team elsewhere in the company; our builds execute on a Linux slave.

Current Attempt

We've got something that kind-of works right now, but it's unreliable. We use 3 build steps:

The first build step is an Execute Shell step with the following commands:

npm install
npm test
node server.js ./test-config.json &

Second we do a Invoke Maven 3 step that points to the test pom.xml.

And third we run Invoke Standalone Sonar Analysis to do static code analysis.

This mostly works, but we depend on Jenkins' ProcessTreeKiller to stop the services once the job completes. We always get the warnings stating: Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+buildfor more information

Unfortunately, we've had cases where the service is terminated too soon (before the tests complete) or where the service doesn't get terminated at all (causing subsequent builds to fail because the port is already in use).

So we need something more reliable.

Failed Attempt

We tried setting up a single shell script which handled starting the service, running maven, killing the service, then outputting an exit code. But this didn't work out because the mvn command wasn't available on the command-line. Our Jenkins has multiple maven versions available (and jdks too) and I don't know where they live on the slaves or how to get at them without using the Invoke Maven 3 build step.

Ideas

We've toyed around with some ideas to solve this problem, but are hoping to get some guidance from others that may have solved similar problems with Jenkins.

1. Have the service self-terminate after some period of time. Problem is figuring out how long to let them run.
2. Add a build step to kill the services after we're done. Problem is that if the maven execution fails, subsequent steps won't run. (And if we tell maven to ignore test failures, then the build doesn't show as broken if they fail.)
3. Try killing any existing service process as the first and last steps of the build. Problem is that other teams also use these Jenkins slaves so we need to make sure that the service is terminated when we're done with our build.
4. Start and stop the node.js services via Maven doing something like this blog suggests. Problem is that we don't know if Jenkins will identify the spawned background task as a "leaked file descriptor" and kill it before we're done testing.

It would be nice if Jenkins had a "Post-build action" that let you run a clean-up script. Or if it had a "Execute background process" build step which would kill the background items at the end of the build. But I can't find anything like that.

Has anyone managed to get Jenkins to do anything remotely like this?

Answer 1

Some brainstorming:

• You can turn off Jenkins ProcessTreeKiller, either globally or per invocation. I am not sure why that is not an option for you.

• In response to #2, several options:

  1. Post-build actions get executed regardless if build steps had failed or not. This would be a great way to trigger a "service cleanup" task that will run regardless of the build state.
  2. You can setup any build step as post-build action, using Any Build Step plugin, or you can use Post Build Tasks plugin, the latter even gives options to define triggering criteria.

• You can change the build state, based on RegEx criteria using Text-finder plugin

• You can setup Conditional Build Steps. The "condition" could even be a result of some script execution