Can consul support large key/value store

Question

We are planning to put our dynamic configuration in hierarchical Consul KV store.

Data is approx 10,000 items and will grow to several thousands as we scale.

we need several nodes (dozens) to wait for updates on the hierarchy root.

Is that a scale that consul is designed to handle ?

thank you

Answer 1

Consul cluster can hold a lot of K/V's. After running several Consul clusters in production we found some things you really want to get correct.

• Make sure you use at least 5 Consul servers in each DC, especially in AWS across availability zones.
• Make sure you set GOMAXPROCS > 1 or you will see poor performance.
• Make sure clients like consul-template are configure to query any server and not just the leader.
• Aggregate documents where you can. If you don't need an individual K/V for each setting in some collection then put them in a document at a single path. It will keep the complexity much lower.

Right now we are running 10, 5 node Consul DCs in production and dev environments.