Merging yaml config files recursively with bash

Question

Is it possible using some smart piping and coding, to merge yaml files recursively? In PHP, I make an array of them (each module can add or update config nodes of/in the system).

The goal is an export shellscript that will merge all separate module folders' config files into big merged files. It's faster, efficient, and the customer does not need the modularity at the time we deploy new versions via FTP, for example.

It should behave like the PHP function: array_merge_recursive

The filesystem structure is like this:

mod/a/config/sys.yml
mod/a/config/another.yml
mod/b/config/sys.yml
mod/b/config/another.yml
mod/c/config/totally-new.yml
sys/config/sys.yml

Config looks like:

date:
   format:
      date_regular: %d-%m-%Y

And a module may, say, do this:

date:
   format:
      date_regular: regular dates are boring
      date_special: !!!%d-%m-%Y!!!

So far, I have:

#!/bin/bash
#........
cp -R $dir_project/ $dir_to/
for i in $dir_project/mod/*/
do
    cp -R "${i}/." $dir_to/sys/
done

This of course destroys all existing config files in the loop.. (rest of the system files are uniquely named)

Basically, I need a yaml parser for the command line, and an array_merge_recursive like alternative. Then a yaml writer to ouput it merged. I fear I have to start to learn Python because bash won't cut it on this one.

Answer 1

I recommend yq -m. yq is a swiss army knife for yaml, very similar to jq (for JSON).

Answer 2

You can use for example perl. The next oneliner:

perl -MYAML::Merge::Simple=merge_files -MYAML -E 'say Dump merge_files(@ARGV)' f1.yaml f2.yaml

for the next input files: f1.yaml

date:
  epoch: 2342342343
  format:
    date_regular: "%d-%m-%Y"

f2.yaml

date:
  format:
    date_regular: regular dates are boring
    date_special: "!!!%d-%m-%Y!!!"

prints the merged result...

---
date:
  epoch: 2342342343
  format:
    date_regular: regular dates are boring
    date_special: '!!!%d-%m-%Y!!!'

Because @Caleb pointed out that the module now is develeloper only, here is an replacement. It is a bit longer and uses two (but commonly available) modules:

• the YAML
• and the Hash::Merge::Simple

perl -MYAML=LoadFile,Dump -MHash::Merge::Simple=merge -E 'say Dump(merge(map{LoadFile($_)}@ARGV))' f1.yaml f2.yaml

produces the same as above.

Answer 3

No.

Bash has no support for nested data structures (its maps are integer->string or string->string only), and thus cannot represent arbitrary YAML documents in-memory.

Use a more powerful language for this task.