Tool to automatically expand YAML merges?

Question

I'm looking for a tool or process which can easily take a YAML file which contains anchors, aliases and merge keys and expand the aliases and merges out into a flat YAML file. There are still many commonly used YAML parses which don't fully support merging.

I'd like to be able to take advantage of merging to keep things DRY, but there are instances where this needs to then be built into a more verbose "flat" YAML file so that it can be used by other tooling which relies on incomplete YAML parsers.

Example Source YAML:

default: &DEFAULT
  URL: website.com
  mode: production  
  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600

development:
  <<: *DEFAULT
  URL: website.local
  mode: dev

test:
  <<: *DEFAULT
  URL: test.website.qa
  mode: test

Desired output YAML:

default:
  URL: website.com
  mode: production  
  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600

development:
  URL: website.local
  mode: dev
  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600

test:
  URL: test.website.qa
  mode: test
  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600

Answer 1

If you have python installed on your system, you can do pip install ruamel.yaml.cmd¹ and then:

yaml merge-expand input.yaml output.yaml

(replace output.yaml with - to write to stdout). This implements the merge expanding with preservation of key order and comments.

The above is actually a few lines of code that utilizes ruamel.yaml¹ so if you have Python (2.7 or 3.4+) and install that using pip install ruamel.yaml and save the following as expand.py:

import sys
from ruamel.yaml import YAML

yaml = YAML(typ='safe')
yaml.default_flow_style=False
with open(sys.argv[1]) as fp:
    data = yaml.load(fp)
with open(sys.argv[2], 'w') as fp:
    yaml.dump(data, fp)

you can already do:

python expand.py input.yaml output.yaml

That will get you YAML that is semantically equivalent to what you requested (in output.yaml the keys of the mappings are sorted, in this programs output they are not).

The above assumes you don't have any tags in your YAML, nor care about preserving any comments. Most of those, and the key ordering, can be preserved by using a patched version of the standard YAML() instance. Patching is necessary because the standard YAML() instance preserves the merges on round-trip as well, which is exactly what you don't want:

import sys
from ruamel.yaml import YAML, SafeConstructor

yaml = YAML()

yaml.Constructor.flatten_mapping = SafeConstructor.flatten_mapping
yaml.default_flow_style=False
yaml.allow_duplicate_keys = True
# comment out next line if you want "normal" anchors/aliases in your output
yaml.representer.ignore_aliases = lambda x: True  

with open(sys.argv[1]) as fp:
    data = yaml.load(fp)
with open(sys.argv[2], 'w') as fp:
    yaml.dump(data, fp)

with this input:

default: &DEFAULT
  URL: website.com
  mode: production
  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600  # an hour?

development:
  <<: *DEFAULT
  URL: website.local     # local web
  mode: dev

test:
  <<: *DEFAULT
  URL: test.website.qa
  mode: test

that will give this output (note that comments on the merged in keys get duplicated):

default:
  URL: website.com
  mode: production
  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600  # an hour?

development:
  URL: website.local     # local web
  mode: dev

  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600  # an hour?

test:
  URL: test.website.qa
  mode: test
  site_name: Website
  some_setting: h2i8yiuhef
  some_other_setting: 3600  # an hour?

The above is what the yaml merge-expand command, mentioned at the start of this answer, does.

---

¹ _{Disclaimer: I am the author of that package.}

Answer 2

UPDATE: 2019-03-13 12:41:05

• This answer was modified pursuant to a comment by Anthon which correctly identified limitations with PyYAML. (See Pitfalls infra).

Context

• YAML file
• Python for parsing the YAML

Problem

• User jtYamlEnthusiast wishes to output a non-DRY version of a YAML file with aliases, anchors, and merge keys.

Solution(s)

• Alternative 1: use the ruamel library promoted by Anthon infra.
• Alternative 2: use Python pprint.pformat and simply do a load/dump round-trip transformation.

Rationale

• the ruamel library is great if you have the discretion to install another python library besides pyyaml, and you want a high degree of control over "round-trip" YAML transformations (such as the preservation of YAML comments, for example).
• if you do not need rigorous control over round-tripped YAML, or you are limited for some other reason to pyyaml, you can simply load and dump YAML directly, in order to obtain the "non-DRY" output.

Pitfalls

• as of this writing PyYAML has limitations relative to the ruamel library, regarding the handling of YAML v1.1 and YAML v1.2

• See also

  • ruamel docs
  • pyyaml repo

Example

    ##
    import pprint
    import yaml
    ##
    myrawyaml = '''
    default: &DEFAULT
      URL: website.com
      mode: production
      site_name: Website
      some_setting: h2i8yiuhef
      some_other_setting: 3600

    development:
      <<: *DEFAULT
      URL: website.local
      mode: dev

    test:
      <<: *DEFAULT
      URL: test.website.qa
      mode: test
    '''
    ##
    pynative  =   yaml.safe_load(myrawyaml)
    vout      =   pprint.pformat(pynative)
    print(vout)                             ##=> this is non-DRY and just happens to be well-formed YAML syntax
    print(yaml.safe_load(vout))             ##=> this proves we have well-formed YAML if it loads without exception