Can't connect to mongodb replicaset via kubectl port-forward

Question

I'm trying to get access to mongodb replicaset via kubectl, so I won't expose it to internet, I can't use OpenVPN since Calico blocks it.

So I'm using this script:

export MONGO_POD_NAME1=$(kubectl get pods --namespace develop -l "app=mongodb-replicaset" -o jsonpath="{.items[0].metadata.name}")
export MONGO_POD_NAME2=$(kubectl get pods --namespace develop -l "app=mongodb-replicaset" -o jsonpath="{.items[1].metadata.name}")
export MONGO_POD_NAME3=$(kubectl get pods --namespace develop -l "app=mongodb-replicaset" -o jsonpath="{.items[2].metadata.name}")
echo $MONGO_POD_NAME1, $MONGO_POD_NAME2, $MONGO_POD_NAME3
kubectl port-forward --namespace develop $MONGO_POD_NAME1 27020:27017 & p3=$!
kubectl port-forward --namespace develop $MONGO_POD_NAME2 27021:27017 & p4=$!
kubectl port-forward --namespace develop $MONGO_POD_NAME3 27022:27017 & p5=$!

wait -n
[ "$?" -gt 1 ] || kill "$p3" "$p4" "$p5"
wait

And my connection string looks like this:

mongodb://LOGIN:PW@localhost:27020,localhost:27021,localhost:27022/animedb?replicaSet=rs0

However, I still can't connect to my mongodb replicaset, it says:

connection error: { MongoNetworkError: failed to connect to server 
[anime-data-develop-mongodb-replicaset-0.anime-data-develop-mongodb-replicaset.develop.svc.cluster.local:27017]
 on first connect [MongoNetworkError: getaddrinfo ENOTFOUND 
anime-data-develop-mongodb-replicaset-0.anime-data-develop-mongodb-replicaset.develop.svc.cluster.local 
anime-data-develop-mongodb-replicaset-0.anime-data-develop-mongodb-replicaset.develop.svc.cluster.local:27017]

But if I use direct connection, I still can connect to each node!
What might be a problem here? How can I connect to mongodb for development?

Answer 1

Port Forwarding will make a local port on your machine redirect (forward) traffic to some pod. In your case, you've asked Kubernetes to forward traffic on 127.0.0.1:27020 to your pod's 27017 port.

The issue happen because the Replica Set configuration points to the other nodes using your internal cluster IPs, so you will see something like [ReplicaSetMonitor-TaskExecutor] changing hosts to rs0/<ClusterIP-1>:27017,<ClusterIP-2>:27017,<ClusterIP-3>:27017 from rs/localhost:27020,localhost:27021,localhost:27022 on your mongo client session, and your machine can't reach your Cluster's IPs, of course.

For development purposes, you'd have to connect to your primary Mongo node only (as in mongodb://localhost:27020/animedb), which will replicate your data into your secondaries. That's safe enough for development/debugging, but not suitable for production!

If you need to set it up for permanent/production access, you should update your replicaSet settings so they find each other using public IPs or hostnames, see https://docs.mongodb.com/manual/tutorial/change-hostnames-in-a-replica-set/.