Unable to properly connect to Redis in Kubernetes

Question

On my macOS (not using Minikube), I have modeled my Kubernetes cluster after this example, which means I have executed this verbatim and in this order:

# Adding my own service to redix-proxy
kubectl create -f ./redis/redis-service.yaml

# Create a bootstrap master
kubectl create -f examples/storage/redis/redis-master.yaml

# Create a service to track the sentinels
kubectl create -f examples/storage/redis/redis-sentinel-service.yaml

# Create a replication controller for redis servers
kubectl create -f examples/storage/redis/redis-controller.yaml

# Create a replication controller for redis sentinels
kubectl create -f examples/storage/redis/redis-sentinel-controller.yaml

# Scale both replication controllers
kubectl scale rc redis --replicas=3
kubectl scale rc redis-sentinel --replicas=3

# Adding my own NodeJS web client server
kubectl create -f web-deployment.yaml

The only difference is in redis-proxy.yaml I used the image image: kubernetes/redis-proxy instead of image: kubernetes/redis-proxy:v2 because I wasn't able to pull the latter.

These are the objects I pass to ioredis to create my Redis instances (one for sessions and one as the main one):

config.js

main: {
  host: 'redis',
  port: 6379,
  db: 5
},
session: {
  host: 'redis',
  port: 6379,
  db: 6
}

---

Error logs:

In my web client web-3448218364-sf1q0 pod, I get this repeated in the logs:

INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event

In my Redis redis-proxy pod, I get this repeated in the logs:

Error connecting to read: dial tcp :0: connection refused

---

Cluster info:

$ kubectl get svc
NAME                      CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
kubernetes                10.91.240.1     <none>           443/TCP          2d
redis                     10.91.251.170   <none>           6379/TCP         31m
redis-sentinel            10.91.250.118   <none>           26379/TCP        31m
web                       10.91.240.16    <none>           80/TCP           31m

$ kubectl get po
NAME                        READY     STATUS    RESTARTS   AGE
redis-2frd0                 1/1       Running   0          34m
redis-master                2/2       Running   0          34m
redis-n4x6f                 1/1       Running   0          34m
redis-proxy                 1/1       Running   0          34m
redis-sentinel-k8tbl        1/1       Running   0          34m
redis-sentinel-kzd66        1/1       Running   0          34m
redis-sentinel-wlzsb        1/1       Running   0          34m
web-3448218364-sf1q0        1/1       Running   0          34m

$ kubectl get deploy
NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
web         1         1         1            1           39m

Question 1) Now, I need to actually connect my application to a Redis pod. I should be connecting to the redis-proxy pod right? So, I created this redis-service.yaml service:

apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - port: 6379
      targetPort: 6379
  selector:
    name: redis-proxy
    role: proxy

I believe I have connected to redis at port 6379 since I usually will get another error message if this is so. Going into the bash shell of my web container web-3448218364-sf1q0, I see the printenv variables of REDIS_SERVICE_PORT=6379 and REDIS_SERVICE_HOST=10.91.251.170.

Question 2) From my error logs, what does it mean by dial tcp :0:? From my interactive Kubernetes console under Services and in the Internal Endpoints column, I see this for the redis service:

redis:6379 TCP
redis:0 TCP

Is this 0 TCP related to that? All of my services have 0 TCP listed in the console, but as you can see, not from the CLI in kubectl get svc.

Answer 1

Always the first thing to check when a kubernetes service does not behave as expected is to check the endpoints of the corresponding service. In your case kubectl get ep redis.

If my assumption is correct it should show you something like this

NAME      ENDPOINTS   AGE
redis     <none>      42d

This means that your service does not select/match any pods.

In your service spec there is the key selector: this selector has to match the labels of the actual deployment you have. You are selecting for all pods with the labels name: redis-proxy and role: proxy which are potentially not matching any pod.

You can run kubectl get pod --show-labels=true to show the labels on the pods and change your service accordingly.

I don't know what the port 0 means in this context. Sometimes it is used to do only DNS resolution with the service.