Deploying stateful applications in Kubernetes

Question

I'm getting acquainted with Kubernetes, and haven't found simple solution for deploying stateful services in Kubernetes.

• Every pod has to be bootstrapped with contact points (list of other pod IPs), which can't be load balanced (i'm afraid of split-brain in case of bad luck: in worst case load-balancer may load-balance every pod to itself, making several self-enclosed clusters of one node)
• Every pod has to have persistent storage that, in worst case, has to be accessed manually (e.g. consul's peers.json)
• Every pod should be reconfigurable; if i've forgot to do something with my consul cluster, rebuilding it from scratch would simply result in downtime. In case kubernetes prevents this, feel free to tell me, i'm still not familiar enough with deployment mechanics.
• Increasing service cluster dynamically with newly-configured instances and then draining older ones may be highly undesired (i'm not consul expert, but from my point of view that drops split-brain protection in consul cluster).

AFAIK the most applicable thing is Pet Set, however, it is still in alpha and can only be deleted completely; also, i don't feel i understand how persistent volumes should be managed to survive Pet Set recreation. Another option i've came up with is splitting service deployments into bootstrap-node deployment, bootstrap-node service and all-other-nodes deployment, which allows me to use bootstrap-node service as a contact point (that's not completely safe, though).

What are the popular approaches for this case and what pros and cons do they have?

Answer 1

If you're looking at a set number of Pods in your stateful cluster in the Kubernetes cluster, PetSets (StatefulSets I believe it is called now) is the answer... or you can define a Service per Pod to achieve the same.

For Pods to be aware of other Pods's IPs, you can use Headless Services, which provide you with the list of IPs associated with a label.

For Storage, if you use emptyDir, you have local storage but you lose it when the Pod is removed / re-scheduled.

I use Zookeeper in Kubernetes and it is a bit of a pain to setup, but Zookeeper provides a 'reconfigure' API which allows to reconfigure the cluster when a node changes, so it is fairly easy to redefine the cluster on startup of a new node when a Pod is rescheduled. I'm not sure if Consul has the same type of feature, but it probably does.