Graceful pod termination

Question

I need to let the container to run 5 minutes after the kubectl ' termination. It needs to do some work before it's destroyed. It seems that kubernetes contains exactly what I need:

terminationGracePeriodSeconds: 300

so I defined it within my yaml. I've updated running RCs, delete current pods so new ones were created and now I can see that a pod contains exactly this setting via get pod xyz -o=yaml.

Unfortunately, when I tried to do rolling-update, the original pod was killed after exactly 1 minute, not after 5 minutes. I does ssh to the target machine and I could see that docker termineted the container after this time.

I tried to do some investigation how the feature works. I finally found the documentation to kubectl delete where there is a notion about graceful termination period:

http://kubernetes.io/docs/user-guide/pods/

By default, all deletes are graceful within 30 seconds. The kubectl delete command supports the --grace-period= option which allows a user to override the default and specify their own value. The value 0 indicates that delete should be immediate, and removes the pod in the API immediately so a new pod can be created with the same name. On the node pods that are set to terminate immediately will still be given a small grace period before being force killed

So I took one pod, nginx, and try to delete it with grace-period=30. It turned out, that original pod was immediately delete and get pods showed that new one was being started.

So no 30 seconds. What am I doing wrong? It seems that all pods kubernetes does not take these values into account. Note that I'm using kubernetes v1.2.2

I also found this issue https://github.com/kubernetes/kubernetes/issues/24695 where the reporter had same problem and he solved it in the same fashion. So e.g. 300 seconds is not too much for kubernetes.

Answer 1

You may be able to set magic sleep in 'preStop' hook.　This hook will be extecuted prior to kubectl sending SIGTERM to your container.

http://kubernetes.io/docs/user-guide/production-pods/#lifecycle-hooks-and-termination-notice

something like:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sleep","300"]