Editing docker-compose.yml with PyYAML

Question

I have a pretty standard docker-compose.yml and I need to edit the password of the database programmatically.

Since its a YAML file, I thought it would be simple to edit and dump the content. So far I tried the PyYAML and it just mess the docker-compose file and I don't know why.

Loading and dumping the same content, it breaks the structure.

Content of the docker-compose.yml:

version: '2'
services:
  web:
    container_name: xxx
    ports:
     - "80:80"
    volumes:
      - .:/xxx
    depends_on:
      - mysql
    build: .
  mysql:
    ports:
     - "32768:32768"
     - "3306:3306"
    container_name: xxx-mysql
    restart: always
    image: mariadb:latest
    environment:
      MYSQL_ROOT_PASSWORD: 'thiswillbechangeonsetupscript'
      MYSQL_DATABASE: 'xxxdb'
    volumes:
     - ./database:/var/lib/mysql
    ports:
      - "3306:3306"

This is how I'm loading and dumping the content:

import yaml

with open("docker-compose.yml", 'r') as ymlfile:
    docker_config = yaml.load(ymlfile)

with open("docker-compose.yml", 'w') as newconf:
    yaml.dump(docker_config, newconf)

And this is how the file is saved.

services:
  mysql:
    container_name: xxx-mysql
    environment: {MYSQL_DATABASE: xxxdb, MYSQL_ROOT_PASSWORD: thiswillbechangeonsetupscript}
    image: mariadb:latest
    ports: ['3306:3306']
    restart: always
    volumes: ['./database:/var/lib/mysql']
  web:
    build: .
    container_name: xxx
    depends_on: [mysql]
    ports: ['80:80']
    volumes: ['.:/xxx']
version: '2'

Is there any better way to do this?! What I'm missing?

Answer 1

The default dump for PyYAML is to use flow style for leaf nodes ( [....] for sequences, {...} for mappings), so the minimum you should do is specify yaml.dump(....., default_flow_style=False)

Then the YAML specifications state that the order of the keys is not guaranteed, and what you see is that PyYAML dumps them in sorted order.

I can recommend to use ruamel.yaml (disclaimer: I am the author of that package), which had the specific goal to allow this kind of round-tripping with minimal, changes compared to the input, often none at all. Including key ordering, flow vs block style, quotes on strings etc.

There is another reason to use ruamel.yaml: if you run this program on your input:

import sys
import ruamel.yaml

yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
yaml.indent(sequence=3, offset=1)

with open("docker-compose.yml", 'r') as ymlfile:
    data = yaml.load(ymlfile)
yaml.dump(data, sys.stdout)

you'll get a DuplicateKeyError:

ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping
  in "docker-compose.yml", line 13, column 5
found duplicate key "ports" with value "[]" (original value: "[]")
  in "docker-compose.yml", line 24, column 5

because ports occurs two times as key in the mapping that is the value for the key mysql. This is not allowed according to the YAML specifications (the old 1.1 that PyYAML fallows and the newer 1.2), but PyYAML silently disposes of the first key-value pair, leaving port 32768 unmapped.

After deleting the last two lines from your input, the output of the program is:

version: '2'
services:
  web:
    container_name: xxx
    ports:
     - "80:80"
    volumes:
     - .:/xxx
    depends_on:
     - mysql
    build: .
  mysql:
    ports:
     - "32768:32768"
     - "3306:3306"
    container_name: xxx-mysql
    restart: always
    image: mariadb:latest
    environment:
      MYSQL_ROOT_PASSWORD: 'thiswillbechangeonsetupscript'
      MYSQL_DATABASE: 'xxxdb'
    volumes:
     - ./database:/var/lib/mysql

which is hopefully close enough for your intended purpose.

Please note that PyYAML remove the quotes in - "80:80", which is fine because 80:80 cannot be falsely interpreted as a sexagesimal, but if you do something with port 25 changing - 80:80 to - 25:25 is vastly different when using a YAML 1.1 parser like PyYAML (as docker-compose does) from - "25:25" (the former equalling- 1525`)

Based on this I have made a utility ruamel.dcw, that uses this functionality to pre-process docker compose files, allowing defaults for environment variables (in case they are not set) and a few other tricks, writes out a temporary file and then calls docker-compose -f tmpfile, you should use a similar technique, disposing of your temporary file after the run.

Answer 2

You need to add default_flow_style=False when you write the yaml:

import yaml

with open("docker-compose.yml", 'r') as ymlfile:
    docker_config = yaml.load(ymlfile)

with open("docker-compose_new.yml", 'w') as newconf:
    yaml.dump(docker_config, newconf, default_flow_style=False)

Then you will get the following as output which, except using alphabetic order to write the lines, is similar to your input:

services:
  mysql:
    container_name: xxx-mysql
    environment:
      MYSQL_DATABASE: xxxdb
      MYSQL_ROOT_PASSWORD: thiswillbechangeonsetupscript
    image: mariadb:latest
    ports:
    - 3306:3306
    restart: always
    volumes:
    - ./database:/var/lib/mysql
  web:
    build: .
    container_name: xxx
    depends_on:
    - mysql
    ports:
    - 80:80
    volumes:
    - .:/xxx
version: '2'

Please note that, in your original docker-compose.yaml you are declaring the ports variable twice, so yaml parser will only consider the last variable. To fix that, remove the following lines:

ports:
  - "3306:3306"

Then, running the write operation as explained above gives the following ouput:

services:
  mysql:
    container_name: xxx-mysql
    environment:
      MYSQL_DATABASE: xxxdb
      MYSQL_ROOT_PASSWORD: thiswillbechangeonsetupscript
    image: mariadb:latest
    ports:
    - 32768:32768
    - 3306:3306
    restart: always
    volumes:
    - ./database:/var/lib/mysql
  web:
    build: .
    container_name: xxx
    depends_on:
    - mysql
    ports:
    - 80:80
    volumes:
    - .:/xxx
version: '2'