kubectl port forwarding timeout issue

Question

While using kubectl port-forward function I was able to succeed in port forwarding a local port to a remote port. However it seems that after a few minutes idling the connection is dropped. Not sure why that is so.

Here is the command used to portforward:

kubectl --namespace somenamespace port-forward somepodname 50051:50051

Error message:

Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
E1125 17:18:55.723715    9940 portforward.go:178] lost connection to pod

Was hoping to be able to keep the connection up

Answer 1

Setting kube's streaming-connection-idle-timeout to 0 should be a right solution, but if you don't want to change anything, you can use while-do construction

Format: while true; do <<YOUR COMMAND HERE>>; done

So just inputing in CLI: while true; do kubectl --namespace somenamespace port-forward somepodname 50051:50051; done should keep kubectl reconnecting on connection lost

Answer 2

Seems there is a 5 minute timeout that can be overridden with kubelet parameters:

https://github.com/kubernetes/kubernetes/issues/19231

If you want to pass something higher than 5 minutes (or unlimited) into your kubelets, you can specify the streaming-connection-idle-timeout. E.g. --streaming-connection-idle-timeout=4h to set it to 4 hours. Or: --streaming-connection-idle-timeout=0 to make it unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

Answer 3

I solved this by keeping the connection alive, e.g. using curl or nc.

Forward the port:

kubectl --namespace somenamespace port-forward somepodname 50051:50051

In another terminal, keep the connection alive by reaching out to the port every 10 seconds:

while true ; do nc -vz 127.0.0.1 50051 ; sleep 10 ; done