kubernetes helm: "lost connection to pod" and "transport is closing" errors

Question

I run helm upgrade --install to modify the state of my kubernetes cluster and I sometimes get an error like this:

22:24:34 StdErr: E0126 17:24:28.472048   48084 portforward.go:178] lost connection to pod
22:24:34 Error: UPGRADE FAILED: transport is closing

It seems that I am not the only one, and it seems to happen with many different helm commands. All of these github issues have descriptions or comments mentioning "lost connection to pod" or "transport is closing" errors (usually both):

• https://github.com/kubernetes/helm/issues/1183
• https://github.com/kubernetes/helm/issues/2003
• https://github.com/kubernetes/helm/issues/2025
• https://github.com/kubernetes/helm/issues/2288
• https://github.com/kubernetes/helm/issues/2560
• https://github.com/kubernetes/helm/issues/3015
• https://github.com/kubernetes/helm/issues/3409

While it can be educational to read through hundreds of github issue comments, usually it's faster to cut to the chase on stackoverflow, and it didn't seem like this question existed yet, so here it is. Hopefully some quick symptom fixes and eventually one or more root cause diagnoses end up in the answers.

Answer 1

Memory limits were causing this error for me. The following fixed it:

kubectl set resources deployment tiller-deploy --limits=memory=200Mi

Answer 2

I was able to correct this by adding the tiller host information to the helm install command.

--host=10.111.221.14:443

You can get your tiller IP this way

$ kubectl get svc -n kube-system tiller-deploy
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
tiller-deploy   ClusterIP   10.111.221.14   <none>        44134/TCP   34h

Full command example

helm install stable/grafana --name=grafana --host=10.111.221.14:4413

I know this is a bit of a work around but all other functions of helm are performing properly after installing via this method. I did not have to add the host information again after the initial install for performing upgrades or rollbacks. Hope this helps!

Answer 3

Deleting the tiller deployment and recreating it is only fix I've seen on github (here and here). This has been most helpful to people when the same helm command fails repeatedly (not with intermittent failures, though you could try it).

delete tiller (helm's server-side component):

kubectl delete deployment -n kube-system tiller-deploy
# deployment "tiller-deploy" deleted

and recreate it:

helm init --upgrade
# $HELM_HOME has been configured at /root/.helm.
# Tiller (the helm server side component) has been upgraded to the current version.
# Happy Helming!

Bouncing tiller obviously won't fix the root cause. There is hopefully a better answer than this forthcoming, maybe from https://github.com/kubernetes/helm/issues/2025. This is the only open github issue as of 13 Feb 2018.