How to fix "process apparently never started in ..." error in Jenkins pipeline?

Question

I am getting the strange error below in my Jenkins pipeline

[Pipeline] withDockerContainer
acp-ci-ubuntu-test does not seem to be running inside a container
$ docker run -t -d -u 1002:1006 -u ubuntu --net=host -v /var/run/docker.sock:/var/run/docker.sock -v /home/ubuntu/.docker:/home/ubuntu/.docker -w /home/ubuntu/workspace/CD-acp-cassandra -v /home/ubuntu/workspace/CD-acp-cassandra:/home/ubuntu/workspace/CD-acp-cassandra:rw,z -v /home/ubuntu/workspace/CD-acp-cassandra@tmp:/home/ubuntu/workspace/CD-acp-cassandra@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** quay.io/arubadevops/acp-build:ut-build cat
$ docker top 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44 -eo pid,comm
[Pipeline] {
[Pipeline] sh
process apparently never started in /home/ubuntu/workspace/CD-acp-cassandra@tmp/durable-70b242d1
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
$ docker stop --time=1 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44
$ docker rm -f 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44
[Pipeline] // withDockerContainer

The corresponding stage in Jenkins pipeline is

    stage("Build docker containers & coreupdate packages") {
        agent {
            docker {
                image "quay.io/arubadevops/acp-build:ut-build"
                label "acp-ci-ubuntu"
                args "-u ubuntu --net=host -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.docker:/home/ubuntu/.docker"
              }
          }
          steps {
              script {
                 try {
                    sh "export CI_BUILD_NUMBER=${currentBuild.number}; cd docker; ./build.sh; cd ../test; ./build.sh;"
                    ciBuildStatus="PASSED"
                 } catch (err) {
                    ciBuildStatus="FAILED"
                 }
              }
          }
      }

What could be the reasons why the process is not getting started within the docker container? Any pointers on how to debug further are also helpful.

Answer 1

The issue is caused by some breaking changes introduced in the Jenkins durable-task plugin v1.31.

Source:

https://issues.jenkins-ci.org/browse/JENKINS-59907 and https://github.com/jenkinsci/durable-task-plugin/blob/master/CHANGELOG.md

Solution: Upgrading the Jenkins durable-task plugin to v1.33 resolved the issue for us.

Answer 2

This error means the Jenkins process is stuck on some command.

Some suggestions:

• Upgrade all of your plugins and re-try.
• Make sure you've the right number of executors and jobs aren't stuck in the queue.
• If you're pulling the image (not your local), try adding alwaysPull true (next line to image).
• When using agent inside stage, remove the outer agent. See: JENKINS-63449.
• Execute org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true in Jenkins's Script Console to debug.
• When the process is stuck, SSH to Jenkins VM and run docker ps to see which command is running.
• Run docker ps -a to see the latest failed runs. In my case it tried to run cat next to custom CMD command set by container (e.g. ansible-playbook cat), which was the invalid command. The cat command is used by design. To change entrypoint, please read JENKINS-51307.
• If your container is still running, you can login to your Docker container by docker exec -it -u0 $(docker ps -ql) bash and run ps wuax to see what's doing.
• Try removing some global variables (could be a bug), see: parallel jobs not starting with docker workflow.

Answer 3

I had this same problem and in my case, it was related to the -u <user> arg passed to the agent. In the end, changing my pipeline to use -u root fixed the problem.

---

In the original post, I notice a -u ubuntu was used to run the container:

docker run -t -d -u 1002:1006 -u ubuntu ... -e ******** quay.io/arubadevops/acp-build:ut-build cat

I was also using a custom user, one I've added when building the Docker image.

agent {
  docker {
    image "app:latest"
    args "-u someuser"
    alwaysPull false
    reuseNode true
  }
}
steps {
  sh '''
    # DO STUFF
  '''
}

Starting the container locally using the same Jenkins commands works OK:

docker run -t -d -u 1000:1000 -u someuser app:image cat
docker top <hash> -eo pid,comm
docker exec -it <hash> ls  # DO STUFF

But in Jenkins, it fails with the same "process never started.." error:

$ docker run -t -d -u 1000:1000 -u someuser app:image cat
$ docker top <hash> -eo pid,comm
[Pipeline] {
[Pipeline] unstash
[Pipeline] sh
process apparently never started in /home/jenkins/agent/workspace/branch@tmp/durable-f5dfbb1c

For some reason, changing it to -u root worked.

agent {
  docker {
    image "app:latest"
    args "-u root"      # <=-----------
    alwaysPull false
    reuseNode true
  }
}

Answer 4

If you have upgraded the durable-task plugin to 1.33 or later and it still won't work, check if there's an empty environment variable configured in your pipeline or stored in the Jenkins configuration (dashed) and remove it: