How to prevent yq removing comments and empty lines?

Question

Here Edit yaml objects in array with yq. Speed up Terminalizer's terminal cast (record) I asked about how to edit yaml with yq. I received the best answer. But by default yq removes comments and empty lines. How to prevent this behavior?

input.yml

# Specify a command to be executed
# like `/bin/bash -l`, `ls`, or any other commands
# the default is bash for Linux
# or powershell.exe for Windows
command: fish -l

# Specify the current working directory path
# the default is the current working directory path
cwd: null

# Export additional ENV variables
env:
  recording: true

# Explicitly set the number of columns
# or use `auto` to take the current
# number of columns of your shell
cols: 110

execute

yq -y . input.yml

result

command: fish -l
cwd: null
env:
  recording: true
cols: 110

Answer 1

In some limited cases you could use diff/patch along with yq.
For example if input.yml contains your input text, the commands

$ yq -y . input.yml > input.yml.1
$ yq -y .env.recording=false input.yml > input.yml.2
$ diff input.yml.1 input.yml.2 > input.yml.diff
$ patch -o input.yml.new input.yml < input.yml.diff

creates a file input.yml.new with comments preserved but recording changed to false:

# Specify a command to be executed
# like `/bin/bash -l`, `ls`, or any other commands
# the default is bash for Linux
# or powershell.exe for Windows
command: fish -l

# Specify the current working directory path
# the default is the current working directory path
cwd: null

# Export additional ENV variables
env:
  recording: false

# Explicitly set the number of columns
# or use `auto` to take the current
# number of columns of your shell
cols: 110

Answer 2

This is improvement of How to prevent yq removing comments and empty lines? comment.

In mine case was no enough diff -B and diff -wB as it still does not keep blank lines and keep generate an entire file difference as a single chunk instead of many small chunks.

Here is example of the input (test.yml):

# This file is automatically generated
#

content-index:

  timestamp: 1970-01-01T00:00:00Z

  entries:

    - dirs:

        - dir: dir-1/dir-2

          files:

            - file: file-1.dat
              md5-hash:
              timestamp: 1970-01-01T00:00:00Z

            - file: file-2.dat
              md5-hash:
              timestamp:

            - file: file-3.dat
              md5-hash:
              timestamp:

        - dir: dir-1/dir-2/dir-3

          files:

            - file: file-1.dat
              md5-hash:
              timestamp:

            - file: file-2.dat
              md5-hash:
              timestamp:

If try to edit a field and generate the difference file:

diff -B test.yml <(yq -y ".\"content-index\".timestamp=\"2022-01-01T00:00:00Z\"" test.yml)

It does keep remove blank lines:

5,7c2
<
<   timestamp: 1970-01-01T00:00:00Z
<
---
>   timestamp: '2022-01-01T00:00:00Z'

Adds everywhere null instead of an empty field and changes the rest of timestamp fields (which means you have to use '...' to retain these as is):

17,19c8,9
<               md5-hash:
<               timestamp: 1970-01-01T00:00:00Z
<
---
>               md5-hash: null
>               timestamp: '1970-01-01T00:00:00+00:00'

The -wB flags changes the difference file from a single chunk into multiple chunks, but still does remove blank lines.

Here is a mention of that diff issue: https://unix.stackexchange.com/questions/423186/diff-how-to-ignore-empty-lines/423188#423188

To fix that you have to use it with grep:

diff -wB <(grep -vE '^\s*$' test.yml) <(yq -y ".\"content-index\".timestamp=\"2022-01-01T00:00:00Z\"" test.yml)

But nevertheless it still does remove comments:

1,2d0
< # This file is automatically generated
< #

Here is solution for that: https://unix.stackexchange.com/questions/17040/how-to-diff-files-ignoring-comments-lines-starting-with/17044#17044

So the complete oneliner is:

diff -wB <(grep -vE '^\s*(#|$)' test.yml) <(yq -y ".\"content-index\".timestamp=\"2022-01-01T00:00:00Z\"" test.yml) | patch -o - test.yml 2>/dev/null

Where 2>/dev/null stands to ignore patch warnings like:

Hunk #1 succeeded at 6 (offset 4 lines).

To avoid it in real code, you can use the -s flag instead:

... | patch -s -o ...

Update:

There is a better complete implementation as a shell script for GitHub Actions pipeline composite action.

GitHub Composite action: https://github.com/andry81-devops/gh-action--accum-content

Bash scripts:

https://github.com/andry81-devops/gh-workflow/blob/master/bash/cache/accum-content.sh
https://github.com/andry81-devops/gh-workflow/blob/master/bash/github/init-yq-workflow.sh

The implementation can use 2 of yq implementations:

• https://github.com/kislyuk/yq - a jq wrapper (default Cygwin distribution)
• https://github.com/mikefarah/yq - Go implementation (default Ubuntu 20.04 distribution)

Search for: yq_edit, yq_diff, yq_patch functions