How to start a service using GitLab runner and prevent it from stopping?

Question

I'm going to deploy simple Spring Boot application using GitLab CI server. My .gitlab-ci.yml is as follows:

stages:
  - build_and_test
  - deploy

web_server_build_and_test:
  stage: build_and_test
  script:
    - mvn clean package

web_server_deploy:
  stage: deploy
  script:
    - mvn clean package -Pprod
    - service gitlab-runner-test stop
    - cp target/*.war /var/gitlab-runner-test/gitlab-runner-test.war
    - chmod +x /var/gitlab-runner-test/gitlab-runner-test.war
    - service gitlab-runner-test start

And the deploy stage produces the following output:

$ service gitlab-runner-test stop
Stopped [13247]
$ cp target/*.war /var/gitlab-runner-test/gitlab-runner-test.war
$ chmod +x /var/gitlab-runner-test/gitlab-runner-test.war
$ service gitlab-runner-test start
Started [21177]

However, I'm unable to load the application since the service has been stopped once runner finishes the stage:

$ service gitlab-runner-test status
Not running (process 21177 not found)

My service scripts delegates actual work to the assembled war package:

#!/usr/bin/env bash

export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre/bin/java
export MODE=service
export APP_NAME=gitlab-runner-test
export PID_FOLDER=/var/run/gitlab-runner-test


/var/gitlab-runner-test/gitlab-runner-test.war $*

Moreover, when I'm starting service by hands (service gitlab-runner-test start) it remains running even after user session has been closed.

I'm not sure, what is the root of the problem - Spring Boot startup script, GitLab configuration, my service scripts or something else?

I'm running Ubuntu 14.04 with GitLab CI multi runner version 0.5.0 (c38415a).

UPD:

Upgrading runner to version 1.0.1 (cffb5c7) does not solve the problem.

Answer 1

Why it's a bad idea to do it...

As its documentation clearly states, the GitLab Runner "runs tests and sends the results to GitLab".

And as tests should start and stop in a timely manner, the runner is designed to kill all created processes after finishing each build.

So it's not a bug that your service is killed, it's a feature. ;)

GitLab CI documentation recommends using dpl for deployment.

dpl is a project that enables you to deploy your app at various PaaS providers, such a Google App Engine, Heroku or Elastic Beanstalk.

So it fires some requests to some REST APIs or pushes some other data through the internets and its process nicely exits.

So doing what you want to do actually requires some hacking - overriding the default runner behaviour. And you should not do it as a long-term solution because it may stop working with some runner/gitlab update.

...but if you insist, then here's the how-to :)

In your case, when you want to actually deploy and run the app on your runner's host itself, we need to use two hacks:

• don't use the default shell runner executor but ssh and make the executor ssh to itself (inspired by michael's solution to this question (please upvote his answer too if upvoting mine!),
• disown the process that is being run by the init script (Joe's solution to this question (again - please upvote it!)

Ok, so here are the instructions:

1. Make sure you can SSH from your runner host to itself with a SSH private key in /root/.ssh/id_rsa, without passphrase, without confirming the fingerprint. ssh localhost run by root should work non-interactively.

2. Edit your gitlab-runner's config file, /etc/gitlab-runner/config.toml to make it look like this:

   [[runners]]
     name = "your-runner-name"
     url = "https://<your_gitlab_instance_fqdn>/ci"
     token = "<your_project_CI_token>"
     tls-ca-file = ""
     executor = "ssh"
     [runners.ssh]
       user = "root"
       password = ""
       host = "localhost"
       port = "22"
       identity_file = "/root/.ssh/id_rsa"
   

(The runner will reload itself after saving the config file)

3. Edit your service script so the process it will create will NOT be a child of the init script and it won't open stdin, stdout and stderr:

   #!/usr/bin/env bash
   
   export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre/bin/java
   export MODE=service
   export APP_NAME=gitlab-runner-test
   export PID_FOLDER=/var/run/gitlab-runner-test
   
   
   /var/gitlab-runner-test/gitlab-runner-test.war $* <&- >&- 2>&- & disown
   

Test by retrying last build or making a commit to your project repo.

---

PS I tested my solution with an init script looking like this:

#!/usr/bin/env bash

start() {
  # Completely disowned process, not a child
  # Credits: Joe at https://stackoverflow.com/a/26420299/2693875
  sleep 99999 <&- >&- 2>&- & disown
  exit 0
}

stop() {
  echo "doing nothing"
  exit 0
}

echo "running on $HOSTNAME..."

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  *)
    echo $"Use this options $0 {start|stop}"
    exit 1
esac

..on Ubuntu 14.04 with gitlab-multi-runner v. 1.02 and GitLab CE 8.5.0.