Jenkinsfile: permission denied when running sh step in Docker container

Question

I have trouble running a simple Jenkinsfile - e.g.

pipeline {
    agent { label 'ssh-slave' } 
    stages {
        stage('Shell Test') {
            steps {
                sh 'echo "Hello World"'
            }
        }
    }
}

The logfiles of Jenkins on the master show that the container was started successfully but the build job crashes with a message like

sh: 1: /home/jenkins/workspace/pipeline@tmp/durable-34c21b81/script.sh: Permission denied

Here are some additional things that we configured / figured out:

1. We are running the agent on a VM with RHEL

2. We are using the Docker Plugin for Jenkins to start / manage the containers on a separate Jenkins agent

3. We are spinning up the Docker container using the Connect with ssh method in the Jenkins plugin and use the jenkinsci/ssh-slave Docker image

4. Jenkins is using the root user in the Docker container (at least all files within /home/jenkins/... are created as root

5. When we add a sleep step into the pipeline and docker exec... into the running container, we cannot execute a simple shell script as root, if we are trying to run it with ./script.sh (even if we set proper file mode with chmod +x script.sh before) - we also get sh: 1: permission denied. But we can run the script, if we use sh script.sh

6. The root user inside the Docker container has a bash - whereas Jenkins is trying to run the script with sh.

7. The error occurs no matter whether we check the run privileged flag in the Docker plugin's template configuration or not

Things we already tried, but didn't work

1. Changing the login shell of the root user in the Docker container to /bin/sh

2. Providing a shebang in the sh step, à la

   sh '''#!/bin/sh
   echo "hello world"
   '''
   

3. Setting the shell executor to /bin/sh in the Jenkins global configuration

4. Changing the Dockerfile of the ssh-slave Docker image in such a way that the ENTRYPOINT does not run a bash script, but runs /bin/sh at the end

Any help is appreciated!

Answer 1

Problem was that /home/jenkins in the container was mounted with noexec:

$ mount
/dev/mapper/rhel-var on /home/jenkins type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)

Underlying issue was that the /var on the underlying host was mounted with noexec (/var is where all the container files reside...):

$ mount
/dev/mapper/rhel-var on /var type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)

So the solution to this problem was to mount /var as executeable on the host via

sudo mount -o remount,exec /var

that solved the issue for us.