Multiple apps running in one kubernetes cluster or a clusters for each app

Question

I have some apps in production working in Azure. All these applications belong to the same company and communicate with each other. I want to migrate them to Kubernetes.

My question is: What are the best practices in this case and why ?

Some peoples recommend one cluster and multiples namespaces and I don't know why.

For example: https://www.youtube.com/watch?v=xygE8DbwJ7c recommends apps within a cluster doing intra-cluster multi-tenancy but the arguments of this choice are not enough for me.

Answer 1

It really depends on the scenario. I can think of one scenario where some of the apps need dedicated higher configuration nodes (Say GPU). In such scenarios having a dedicated cluster with GPU nodes can be beneficial for such apps. And having a normal CPU nodes for other normal apps.

Answer 2

My question is: What are the best practices in this case? and why ?

Answer is: it depends...

To try to summarize it from our experience:

• Cluster for each app is usually quite a bit waste of resources, especially giving HA clusters requirements, and it can mainly be justified in case when single app is comprised of larger number of microservices that are naturally clustered together or when some special security considerations has to be taken into account. That is, however, in our experience, rare the case (but it depends)...

• Namespaces for apps in a cluster are more in line with our experience and needs, but again, this should not be overdone either (so, again it depends) since, for example your CNI can be bottleneck leading to one rogue app (or setup) degrading performance for other apps in seemingly unrelated case. Loadbanalcing and rollout downtimes, clashes for resources and other things can happen if all is crammed into one cluster at all cost. So this has it's limits as well.

• Best of both worlds - we started with single cluster, and when we reached naturally separate (and separately performant) use cases (say, qa, dev, stage environments, different client with special security considerations etc) we migrated to more clusters, keeping in each cluster reasonably namespaced apps.

So all in all: depending on available machine pool (number of nodes), size of the cluster, size of apps themselves (microservice/service complexity), HA requirements, redundance, security considerations etc.... you might want to fit all into one cluster with namespaced apps, then again maybe separate in several clusters (again with namespaced apps within each cluster) or keep everything totally separate with one app per cluster. So - it depends.