Docker service won't run inside Container - ERROR : ulimit: error setting limit (Operation not permitted)

Question

I'm running a cluster on AWS EKS. Container(StatefulSet POD) that currently running has Docker installation inside of it.

I ran this image as Kubernetes StatefulSet in my cluster. Here is my yaml file,

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
  labels:
    run: jenkins
spec:
  serviceName: jenkins
  replicas: 1
  selector:
    matchLabels:
      run: jenkins
  template:
    metadata:
      labels:
        run: jenkins
    spec:
      securityContext:
        fsGroup: 1000
      containers:
      - name: jenkins
        image: 99*****.dkr.ecr.<region>.amazonaws.com/<my_jenkins_image>:0.0.3
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: jenkins-port

Inside this POD, I can not run any docker command which gives a ERROR:

/etc/init.d/docker: 96: ulimit: error setting limit (Operation not permitted)

In my research, I went through some artcile which did not fix my issue. I have listed down solution that i tried but not fixed in my case

First solution: (I ran inside the container) aricle link

$ sudo service docker stop
$ sudo bash -c "echo \"limit nofile 262144 262144\" >> /etc/init/docker.conf"
$ sudo service docker start

Second solution: (I ran inside the container)

ulimit -n 65536 in /etc/init.d/docker

Third solution: ** article link This seems a far better answer, which i could not add into my configuration file. it says, run pod with as privilaged. But there is no way to add that option in ***Kubernetes StatefulSet* . So I tried by adding a SecurityContext (securityContext:fsGroup: 1000) like this inside configuration file,

spec:
  serviceName: jenkins
  replicas: 1
  selector:
    matchLabels:
      run: jenkins
  template:
    metadata:
      labels:
        run: jenkins
    spec:
      securityContext:
        fsGroup: 1000

still it does not work.

Note :same image worked on Docker swarm

Anyhelp would be appreciated!

Answer 1

I had this issue with Elastic Search and adding initContainer worked. In this case it could be the solution:

spec:
  .
  .
  .
  initContainers:  
  - name: increase-fd-ulimit
    image: busybox
    command: ["sh", "-c", "ulimit -n 65536"]
    securityContext:
      privileged: true

If it doesn't work, there is a second way to solve this problem which includes creating a new Dockerfile or changing existing:

FROM 99*****.dkr.ecr.<region>.amazonaws.com/<my_jenkins_image>:0.0.3
RUN ulimit -n 65536
USER 1000

and change securityContext to:

  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    capabilities:
      add: ["IPC_LOCK"]