Load .gitlab-ci.yml with pyyaml fails with could not determine constructor

Question

Loading .gitlab-ci.yml fails when I try to load it with yaml.load.

yaml.constructor.ConstructorError: could not determine a constructor for the tag '!reference'
  in "python.yml", line 9, column 7

That's the yaml I try to load.

.python:
  before_script:
    - pip install -r requirements.txt
  script:
    - echo "Hello Python!"

test:
  before_script:
    - !reference [.python, before_script]
  script:
    - pytest

Right now I'm not interested in those references. However, I'm modifying some parts of the yaml and then write it back to the filesystem. So, I don't want to strip these references.

• Are there any GitLab libraries to load gitlab-ci.yml? I could not find any.
• Is there a way to treat !reference [.python, before_script] as String and keep it as is?

Answer 1

Upgrade to ruamel.yaml>=0.15.x

    yaml_str = """
.python:
  before_script:
    - pip install -r requirements.txt
  script:
    - echo "Hello Python!"

test:
  before_script:
    - !reference [.python, before_script]
  script:
    - pytest """

from ruamel.yaml import YAML
yaml = YAML()
yaml.preserve_quotes = True
data = yaml.load(yaml_str)

Answer 2

In the meantime, I have figured out how to add a constructor and representer to load and dump YAML with custom tags.

import yaml

class Reference:
    yaml_tag = u'!reference'
    def __init__(self, values):
        self._values = values

def reference_constructor(loader: yaml.SafeLoader, node: yaml.nodes.CollectionNode) -> Reference:
    return Reference(loader.construct_sequence(node))

def reference_representer(dumper: yaml.SafeDumper, ref: Reference) -> yaml.nodes.SequenceNode:
    return dumper.represent_sequence(Reference.yaml_tag, ref._values)

def loader():
    loader = yaml.SafeLoader
    loader.add_constructor(Reference.yaml_tag, reference_constructor)
    return loader

def dumper():
    dumper = yaml.SafeDumper
    dumper.add_representer(Reference, reference_representer)
    return dumper

def rewrite(ci_file):
    with open(ci_file) as f:
        ci_yaml = yaml.load(f, Loader=loader())
    print(ci_yaml)
    with open(ci_file, "w") as f:
        yaml.dump(ci_yaml, f, Dumper=dumper())

if __name__ == "__main__":
    rewrite(".gitlab-ci.yml")