Comparing HA-cluster platforms for my usecase

Question

I have been trying Docker Swarm and looking into other solutions, such as Kubernetes, but I just can't figure out what would be the best for my use-case, and I could use some help from experts, so your input is very welcome.

I have some requirements for the cloud I want to build, and it (obviously) should be done as cheap, simple and reliable as possible:

• Host stateless containers, such as web containers, with a production-quality loadbalancer with automatic HTTPS (Let's Encrypt)
• Host stateful containers, such as MySQL, in both a non-clustered approach (so: I have only one replica (because I don't need scaling there), but if that server fails, it would be nice if that container moves to another host automatically, without data loss) and in a clustered approach (with Galeria for instance).
  • These databases need fast storage, so preferably they would store their stuff locally at first, and when they move, the volume moves with them.
• Share volumes between hosts automatically with the fastest diskspeed possible for an open source solution.
• Preferably stay with my current datacenter, where I host my VPSes (I have three of them, connected with 10GBit links) on SSD storage
  • Explanation: I want to stay with my current datacenter because I like their support, they have easy automatic (non-credit card, because I'm running a Dutch company) payment, they are affordable and they have great certifications (such as ISO 27001:2013, PCI DSS, ISO 9001:2008, NEN 7510 etc).
  • You may suggest moving to Azure, AWS or GCP, but I would rather not.
  • I don't mind building my own cluster, as long as it's doable, alone, at (relative) production quality.

Keep in mind that I'm not running a lot of services (only 10-20), but I do need production quality and high-availability.

Also, I prefer to run things that normally aren't run in containers, in the container ecosystem anyway to have more flexibility and having them be restarted when a host fails automatically.

---

Things I have already considered:

• Running Docker Swarm, with Traefik and Gluster: Traefik seems stable (most of the time), automatically get certificates, and Gluster can be used - easily - to share volumes. However: there is no supported production quality Gluster volume driver, and thus I have to use bind mounts, and I run into permission issues.
• Running Kubernetes with some ingress controller (for instance cert-manager, but that's not for production, or Traefik, or Voyager?) and the Gluster system for storing volumes, but Kubernetes seems overcomplicated.
• Going to AWS or GCP anyway, but their pricing is confusing and they pay by credit card.
• Using something like Flynn, because I only have web applications anyway, and the Heroku style system seems fun, but they still have no HTTPS support in a stable release: https://github.com/flynn/flynn/issues/1995

Answer 1

Honestly, from a pure ROI and operational load perspective, it doesn't sound like container runtimes at your current datacenter is the right solution for this problem.

From an operator perspective, a container system makes sense when there are tenancy and heterogeneity problems that are hard to solve with VMs/VPSes, and the plant is at minimum dozens of nodes in size. Running any container infrastructure in an HA manner is a lot of work, and there are a lot of corner cases that require dedicated, specialist attention. The need has to be large enough for it to make sense to make this investment.

The plant as described, with redundancy, can run on a handful/dozen VMs/VPSes. It needs some careful architecting to achieve desired levels of availability, but the patterns for managing databases and stateless apps on VMs for HA with, say, 3x scalability, are pretty well established.

There is still a lot of discovery happening in the container world. With Kubernetes especially, every quarter there is a whole new release with new corner cases to discover.

Of course, it's really fun to learn about it, but it's still at the state where it's marvelous to see it working, not boring.