HashiCorp Vault - Setup / Architecture in Production

Question

I'm getting ready to setup HashiCorp Vault with my web application, and while the examples HashiCorp provides make sense, I'm a little unclear of what the intended production setup should be.

In my case, I have:

• a small handful of AWS EC2 instances serving my web application
• a couple EC2 instances serving Jenkins for continuous deployment

and I need:

• My configuration software (Ansible) and Jenkins to be able to read secrets during deployment
• to allow employees in the company to read secrets as needed, and potentially, generate temporary ones for certain types of access.

I'll probably be using S3 as a storage backend for Vault.

The types of questions I have are:

1. Should vault be running on all my EC2 instances, and listening at 127.0.0.1:8200?

2. Or do I create an instance (maybe 2 for availability) that just run Vault and have other instances / services connect to those as needed for secret access?

3. If i needed employees to be able to access secrets from their local machines, how does that work? Do they setup vault locally against the S3 storage, or should they be hitting the REST API of the remote servers from step 2 to access their secrets?

4. And to be clear, any machine that's running vault, if it's restarted, then vault would need to be unsealed again, which seems to be a manual process involving x number of key holders?

Answer 1

Vault runs in a client-server architecture, so you should have a dedicated cluster of Vault servers (usually 3 is suitable for small-medium installations) running in availability mode.

The Vault servers should probably bind to the internal private IP, not 127.0.0.1, since they they won't be accessible within your VPC. You definitely do not want to bind 0.0.0.0, since that could make Vault publicly accessible if your instance has a public IP.

You'll want to bind to the IP that is advertised on the certificate, whether that's the IP or the DNS name. You should only communicate with Vault over TLS in a production-grade infrastructure.

Any and all requests go through those Vault servers. If other users need to communicate with Vault, they should connect to the VPC via a VPN or bastion host and issue requests against it.

When a machine that is running Vault is restarted, Vault does need to be unsealed. This is why you should run Vault in HA mode, so another server can accept requests. You can setup monitoring and alerting to notify you when a server needs to be unsealed (Vault returns a special status code).

You can also read the production hardening guide for more tips.