store and retrieve files from hashicorp vault

Question

I can't figure out how to store files in hashicorp vault. Our use case for a PoC is to store a SSL cert at a certain path and then download it via the HTTP API.

I tried using the kv secrets engine which seems the most appropriate.

Answer 1

It seems that you can specify a file with data in it to store as the value for a key in HashiCorp vault.

You can use

vault write <path> -value=@file to write the contents of file to the key specified in path.

So if you want to store the contents of a crt you can do:

vault write secret/ssl-certs/prod-1 [email protected]

One thing to keep in mind is that you're not saving the file but the contents of the file.

Answer 2

while loading kv pairs in vault you can also load using a json file with one of the keys as the cert. Below is a sample cert generated at https://www.digicert.com/order/sample-csr.php

-----BEGIN CERTIFICATE REQUEST-----
MIICvDCCAaQCAQAwdzELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFV0YWgxDzANBgNV
BAcMBkxpbmRvbjEWMBQGA1UECgwNRGlnaUNlcnQgSW5jLjERMA8GA1UECwwIRGln
aUNlcnQxHTAbBgNVBAMMFGV4YW1wbGUuZGlnaWNlcnQuY29tMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8+To7d+2kPWeBv/orU3LVbJwDrSQbeKamCmo
wp5bqDxIwV20zqRb7APUOKYoVEFFOEQs6T6gImnIolhbiH6m4zgZ/CPvWBOkZc+c
1Po2EmvBz+AD5sBdT5kzGQA6NbWyZGldxRthNLOs1efOhdnWFuhI162qmcflgpiI
WDuwq4C9f+YkeJhNn9dF5+owm8cOQmDrV8NNdiTqin8q3qYAHHJRW28glJUCZkTZ
wIaSR6crBQ8TbYNE0dc+Caa3DOIkz1EOsHWzTx+n0zKfqcbgXi4DJx+C1bjptYPR
BPZL8DAeWuA8ebudVT44yEp82G96/Ggcf7F33xMxe0yc+Xa6owIDAQABoAAwDQYJ
KoZIhvcNAQEFBQADggEBAB0kcrFccSmFDmxox0Ne01UIqSsDqHgL+XmHTXJwre6D
hJSZwbvEtOK0G3+dr4Fs11WuUNt5qcLsx5a8uk4G6AKHMzuhLsJ7XZjgmQXGECpY
Q4mC3yT3ZoCGpIXbw+iP3lmEEXgaQL0Tx5LFl/okKbKYwIqNiyKWOMj7ZR/wxWg/
ZDGRs55xuoeLDJ/ZRFf9bI+IaCUd1YrfYcHIl3G87Av+r49YVwqRDT0VDV7uLgqn
29XI1PpVUNCPQGn9p/eX6Qo7vpDaPybRtA2R7XLKjQaF9oXWeCUqy1hvJac9QFO2
97Ob1alpHPoZ7mWiEuJwjBPii6a9M9G30nUo39lBi1w=
-----END CERTIFICATE REQUEST-----

In order to store the above cert and saving it as a key value pair in a json file the newlines will have to be replaced with \n to save it as a single continuous string Below is the content of the json file (with the same cert saved as the value ) vault_certfile_kv_stackoverflow.json

{
"sample.ssl.public.cert":"-----BEGIN CERTIFICATE REQUEST-----\nMIICvDCCAaQCAQAwdzELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFV0YWgxDzANBgNV\nBAcMBkxpbmRvbjEWMBQGA1UECgwNRGlnaUNlcnQgSW5jLjERMA8GA1UECwwIRGln\naUNlcnQxHTAbBgNVBAMMFGV4YW1wbGUuZGlnaWNlcnQuY29tMIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8+To7d+2kPWeBv/orU3LVbJwDrSQbeKamCmo\nwp5bqDxIwV20zqRb7APUOKYoVEFFOEQs6T6gImnIolhbiH6m4zgZ/CPvWBOkZc+c\n1Po2EmvBz+AD5sBdT5kzGQA6NbWyZGldxRthNLOs1efOhdnWFuhI162qmcflgpiI\nWDuwq4C9f+YkeJhNn9dF5+owm8cOQmDrV8NNdiTqin8q3qYAHHJRW28glJUCZkTZ\nwIaSR6crBQ8TbYNE0dc+Caa3DOIkz1EOsHWzTx+n0zKfqcbgXi4DJx+C1bjptYPR\nBPZL8DAeWuA8ebudVT44yEp82G96/Ggcf7F33xMxe0yc+Xa6owIDAQABoAAwDQYJ\nKoZIhvcNAQEFBQADggEBAB0kcrFccSmFDmxox0Ne01UIqSsDqHgL+XmHTXJwre6D\nhJSZwbvEtOK0G3+dr4Fs11WuUNt5qcLsx5a8uk4G6AKHMzuhLsJ7XZjgmQXGECpY\nQ4mC3yT3ZoCGpIXbw+iP3lmEEXgaQL0Tx5LFl/okKbKYwIqNiyKWOMj7ZR/wxWg/\nZDGRs55xuoeLDJ/ZRFf9bI+IaCUd1YrfYcHIl3G87Av+r49YVwqRDT0VDV7uLgqn\n29XI1PpVUNCPQGn9p/eX6Qo7vpDaPybRtA2R7XLKjQaF9oXWeCUqy1hvJac9QFO2\n97Ob1alpHPoZ7mWiEuJwjBPii6a9M9G30nUo39lBi1w=\n-----END CERTIFICATE REQUEST-----"
}

finally here is how to upload this json file

vault write --address=https://<vaultdomain> secret/<path> @vault_certfile_kv_stackoverflow.json

Answer 3

So Vault's Default offering doesn't have this baked in, but there's a Desktop GUI program that add's this functionality in a user friendly way.
https://github.com/adobe/cryptr
I did run into a bit of confusion when using it:
If you have a KVv2, the HC Web UI, and the Cryptr Desktop GUI will use different conventions.
When writing Vault policies you'd use /KVv2/data/path/
When using Cryptr you'd use /KVv2/data/path/
When using HC WebUI you'd use /kvv2/path/

Fact: You can utilize base64 encoding to store raw binary files in any KV store.
Thus you can use the technique to store in Hashicorp Vault as well.

So base64 encoding is a reversible function that allows you to take any binary file, convert it to a 1 line string, then take the generated 1 line string and convert it back to any binary file. And since you can store a 1 line string in any KV store, you can store arbitrary binary files in any KV store! :) (*)

Here's some code to do what you're asking:

CMD:\> vault server -dev
WindowsSubsystemForLinuxBash:/mnt/c# curl -L https://releases.hashicorp.com/vault/1.0.2/vault_1.0.2_linux_amd64.zip > vault.zip
Bash# apt-get update
Bash# apt-get install unzip
Bash# unzip vault.zip -d /bin
Bash# chmod +x /bin/vault
Bash# export VAULT_ADDR=http://127.0.0.1:8200
Bash# vault login s.aO8ustaAV4Ot1OxzBe94vi3J
Bash# cat excelfile.xlsx | md5sum
fb6b4eaa2be1c8c410645a5f0819539e  -
Bash# cat excelfile.xlsx | base64 | base64 --decode > x.xlsx
Bash# cat x.xlsx | md5sum
fb6b4eaa2be1c8c410645a5f0819539e  -
Bash:/mnt/c# cat excelfile.xlsx | base64 | vault kv put secret/excelfile.xlsx base64dfile=-
(=- means assign value from standard in, which in this case is the piped output of the cat file command)
Chrome: localhost:8200
(login with dev root token, and you'll see the value is characters in a 1 line string)
Bash# rm excelfile.xlsx
Bash# vault kv get -field=base64dfile secret/excelfile.xlsx | tr -d '\n' | base64 --decode > excelfile.xlsx
(or)
Bash# vault kv get -field=base64dfile secret/excelfile.xlsx | sed 's/\r//' | base64 --decode > excelfile.xlsx
Bash# cat excelfile.xlsx | md5sum
fb6b4eaa2be1c8c410645a5f0819539e  -

(*Note Vault and other KV stores often have file size limits, Vault with Consul backend would have a secret file size limit of around ~375kb since base64 encoding will bloat the file size by 4/3rds bringing the size to 500kb and Consul has a Key Value pair limit of 0.5mb ish.)
(Note for perspective that's plenty of space as cert files can be ~8KB/if it's larger than 375kb it's probably not a secret.)

Lets say down the road you need to store bigger secrets:
(Such as Kubernetes etcd snapshot)
Since Vault went 1.0, there's built in functionality to migrate your storage backend, so you could switch from "Consul Storage Backend" to "Hybrid Storage Backend of AWS S3 Storage with Consul" (Consuls still needed for HA consistency locking in multi server setups)" to have a bigger limit. Picking a different storage backend will give you a bigger KV size limit. Note Vault probably imposes a sensible limit like 10mb though. Because even if you had a Vault Backend that supported 1TB Key Value sizes, you definitely would want to think twice about storing large files in vault because the base64 process will add computing overhead as well as bloat the files by 4/3rds so a 300mb file would take up 400mb of space once base64'd. (That being said it could make since for the sake of consistency, consistency is good for automation and maintainability, and compute/storage resources.)

Here's how I'd use Vault if I needed to support large secrets:
I'd write a wrapper python script to get and fetch secrets from vault, and I'd have 3 scenarios, 2 reserved keywords, and the following naming convention/logic:

1. For secrets > 375kb
   secret/filename bigfile:json containing a symmetric encryption key and the location of an encrypted file stored in a spot designed to store large files.
   Wrapper script would recognize "bigfile" as a reserved keyword, and execute logic to parse json, download encrypted file from file store (Torrent/TFP server/CephFS Path/Azure Blob/AWS S3/GCP Cloud Storage), and decrypt the file a for me to my current context.
2. For secret binary files < 375kb
   secret/filename base64dfile:1 line string of characters representing the base64 encoded version of the binary file
   Wrapper script would recognize "base64dfile" as a reserved keyword, and execute logic to unbase64 it and convert to a file upon fetching it.
3. For text files (.json with secrets, .yamls with secrets, .pem certs, etc) < 375
   secret/filename filename:filecontents as multiline strings are allowed