How to insert a comment line to YAML in Python using ruamel.yaml?

Question

I have a structure like this to which I want to add comment lines using ruamel.yaml:

xyz:
  a: 1    # comment 1
  b: 2

test1:
  test2:
    test3: 3

Now, I want to insert comment-lines (not eol_comments) to make it look like this:

xyz:
  a: 1    # comment 1
  b: 2

# before test1 (top level)
test1:
  # before test2
  test2:
    # after test2
    test3: 3

I know, that I can add eol_comments using ruamel.yaml, but I didn't find a way to add whole comment lines.

Answer 1

There is indeed not a function in ruamel.yaml<=0.12.18 to insert a comment on the line before a key, but there is a function to set a comment at beginning of a structure: .yaml_set_start_comment. With that you could already set two of the three comments you want to add:

import sys
import ruamel.yaml

yaml_str = """\
xyz:
  a: 1    # comment 1
  b: 2

test1:
  test2:
    test3: 3
"""

data = ruamel.yaml.round_trip_load(yaml_str)
data['test1'].yaml_set_start_comment('before test2', indent=2)
data['test1']['test2'].yaml_set_start_comment('after test2', indent=4)
ruamel.yaml.round_trip_dump(data, sys.stdout)

gives:

xyz:
  a: 1    # comment 1
  b: 2

test1:
  # before test2
  test2:
    # after test2
    test3: 3

There is actually a "comment" constituting of the empty line between the value for xyz and test1, but if you append your comment to that structure and then insert a new key before test1 things don't show up as you want. Therefore the thing to do is insert the comment explicitly before key test1. You can round-trip load your expected output to see what the internal Comment should look like:

yaml_str_out = """\
xyz:
  a: 1    # comment 1
  b: 2

# before test1 (top level)
test1:
  # before test2
  test2:
    # before test3
    test3: 3
"""
test = ruamel.yaml.round_trip_load(yaml_str_out)
print(test.ca)

gives (wrapped this for easier viewing):

Comment(comment=None,
        items={'test1': [None, 
                        [CommentToken(value='# before test1 (top level)\n')], 
                        None, 
                        [CommentToken(value='# before test2\n')]]})

As you see # before test2 is considered to be a comment after the key. And doing test['test1'].yaml_set_start_comment('xxxxx', indent=2) will not have any effect as the comment associated with test1 overrules that and # xxxxx will not show up in a dump.

With that information and some background knowledge, I adapted some of the code from yaml_set_start_comment() (assuming the original imports and yaml_str):

def yscbak(self, key, before=None, indent=0, after=None, after_indent=None):
    """
    expects comment (before/after) to be without `#` and possible have multiple lines
    """
    from ruamel.yaml.error import Mark
    from ruamel.yaml.tokens import CommentToken

    def comment_token(s, mark):
        # handle empty lines as having no comment
        return CommentToken(('# ' if s else '') + s + '\n', mark, None)

    if after_indent is None:
        after_indent = indent + 2
    if before and before[-1] == '\n':
        before = before[:-1]  # strip final newline if there
    if after and after[-1] == '\n':
        after = after[:-1]  # strip final newline if there
    start_mark = Mark(None, None, None, indent, None, None)
    c = self.ca.items.setdefault(key, [None, [], None, None])
    if before:
        for com in before.split('\n'):
            c[1].append(comment_token(com, start_mark))
    if after:
        start_mark = Mark(None, None, None, after_indent, None, None)
        if c[3] is None:
            c[3] = []
        for com in after.split('\n'):
            c[3].append(comment_token(com, start_mark))

if not hasattr(ruamel.yaml.comments.CommentedMap, 
               'yaml_set_comment_before_after_key'):
    ruamel.yaml.comments.CommentedMap.yaml_set_comment_before_after_key = yscbak


data = ruamel.yaml.round_trip_load(yaml_str)
data.yaml_set_comment_before_after_key('test1', 'before test1 (top level)',
                                       after='before test2', after_indent=2)
data['test1']['test2'].yaml_set_start_comment('after test2', indent=4)
ruamel.yaml.round_trip_dump(data, sys.stdout)

and get:

xyz:
  a: 1    # comment 1
  b: 2

# before test1 (top level)
test1:
  # before test2
  test2:
    # after test2
    test3: 3

The test with hasattr is to make sure you don't overwrite such a function when it gets added to ruamel.yaml

BTW: All comments are end-of-line comments in YAML, there might just be valid YAML before some of those comments.