Hosting Jenkins on Kubernetes while using docker.sock

Question

I'm trying to host a Jenkins image on GKE to run a build. Mostly, I've followed Google's tutorial for setting up Jenkins in Kubernetes. I've got a fairly basic set-up with one master node which runs the builds.

I also want to be able to use Docker inside of the Jenkins environment, and so I've gone into Jenkins' Global Tools Configuration and added a Docker instance. I've additionally mapped the docker.sock in my deployment file to bypass a "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" problem.

My current deployment looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: master
    spec:
      containers:
      - name: master
        image: jenkins/jenkins:2.95
        ports:
        - containerPort: 8080
        - containerPort: 50000
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 2
          failureThreshold: 5
        env:
        - name: JENKINS_OPTS
          valueFrom:
            secretKeyRef:
              name: jenkins
              key: options
        - name: JAVA_OPTS
          value: '-Xmx1400m'
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-home
        - mountPath: /var/run/docker.sock
          name: docker-socket
        securityContext:
          privileged: true
        resources:
          limits:
            cpu: 500m
            memory: 1500Mi
          requests:
            cpu: 500m
            memory: 1500Mi
      volumes:
      - name: jenkins-home
        gcePersistentDisk:
          pdName: jenkins-home
          fsType: ext4
          partition: 1
      - name: docker-socket
        hostPath:
          path: /var/run/docker.sock

Unfortunately, any builds fail with the following error:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: 
....
dial unix /var/run/docker.sock: connect: permission denied

Most Google searches involving this error seem don't seem to be Kubernetes-related.

What am I missing?

Update: To some extent, this works better if I use this configuration:

spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: master
    spec:
      securityContext:
        runAsUser: 0
      containers:

which basically runs Jenkins as root. Unfortunately, that has some consequences for the way that pipelines manage auth credentials. When I try to use "withRepository(repoName, credentialId)", the pipeline adds an entry to /var/jenkins_home/.dockercfg, but a later docker push step doesn't seem to be able to find those credentials (I think that later step ends up looking in /root/.dockercfg or /root/.docker/config.json).

Answer 1

You might want to try running it as the 1000 user:

...
spec:
  ...
  securityContext:
    # Specify fsGroup for pod, so that the persistent volume is writable for the non-privileged uid/gid 1000
    runAsUser: 1000
    fsGroup: 1000
  ...

You may also find this Helm chart useful.

Answer 2

I hope this helps someone who faced this problem and changing volume type to file of runAsUser to 1000 didn't work.

For me setting the runAsUser : 0 worked for me.

I am not sure if this is adviced as per this post. But you can try to set the DOCKE_HOST env variable and check if that works for you as well.