Can a kubernetes pod dynamically change its own label?

Question

I'm trying to setup a postgresql cluster in primary-standby mode using repmgr, and I'm thinking about how to correctly redirect the traffic to the "primary" pod.

My intuition is to use one label to "highlight" the primary pod and rely on the label selector of the service object to "bind" the cluster IP to it. But this lead to the question: How to "move" the label from the old primary pod to a new one after the failover?

Is there a way like let me register a custom script in my pod definition yaml to periodically check the role of pod and change the pod label depending on results?

Do you guys know if this method is possible? Or maybe there is already a sharp solution that can deal with my situation?

Answer 1

• StatefulSets is the answer!

I'm quoting from offical Kubernetes Docs

Like a Deployment , a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

And even more

StatefulSets are valuable for applications that require one or more of the following.

1. Stable, unique network identifiers.
2. Stable, persistent storage.
3. Ordered, graceful deployment and scaling.
4. Ordered, automated rolling updates.

I believe, the point number 1 is more important for you(According to your question). StatefullSets maintain order of spawing the pods too. Back in a while I worked on Redis Cluster deployment on K8s, I don't remember correctly or repo, but they used some scripts in the container to determine the Redis Master. Since StatefullSet maintain the order of spawning, it is easy to make first spawned pod is master. And again, please refer the doc till the end and any blogs on this concept.

And even even more

Stable Network ID

Each Pod in a StatefulSet derives its hostname from the name of the StatefulSet and the ordinal of the Pod. The pattern for the constructed hostname is $(statefulset name)-$(ordinal)

Because of stable n/w ID, the DNS name for pod never changes

I would recommend to deploy your postgress cluster as StatefullSet. Please refer below links to get some idea

• https://kubernetes.io/blog/2017/02/postgresql-clusters-kubernetes-statefulsets/
• https://portworx.com/ha-postgresql-kubernetes/
• Or deploy as helm charts - https://github.com/helm/charts/tree/master/stable/postgresql