How to have base values in pillars that can be overridden?

Question

I would like to store all Salt files (pillars, states, data files, etc.) in a git repository, so that this repository can be cloned on several different deployments.

Then I would like to be able to change the value of some pillar settings, such as a pathname, or a password, but without editing the original file which is in version control (i.e. those modifications would be local only and not necessarily versioned).

I would like to be able to pull new versions from the original repository (e.g. to add new pillar and state definitions) without losing the customized values.

E.g. the "base" or "default" pillar file would have settings like:

service:
  dir: /var/opt/myservice
  username: myuser
  password: mypassword

and I would like to customize some settings, in another file, without changing the base file:

service:
  dir: /mnt/data/myservice
  password: secret_password

The modified settings should take precedence over the base / default ones.

Is it possible to do this by using environments (e.g. a "base" environment and a "custom" environment)?

Or perhaps by including these custom pillar files?

The documentation seems to indicate that there isn't a fixed order for overriding pillar settings.

Answer 1

Let me first suggest a way where you keep the original file and the customized settings in the git repository. See below how to override setting with a file outside of git.

Setup Git Pillar

I assume all files are stored in a git pillar like described here. I am using the syntax of salt version 2015.8 here.

ext_pillar:
  - git:
    - master https://gitserver/git-pillar.git:
      - env: base

In your top.sls file you can list different SLS files. They will override each other in the order listed in the top file:

# top.sls
base:
  '*':
     standard
  '*qa'
     qaservers
  'hostqa':
     hostqaconfig

This will apply on all servers:

# standard.sls
test:
  setting1: A
  setting2: B

This will apply on all servers with the name ending with 'qa':

# qaservers.sls
test:
  setting2: B2

This will apply to the server with the name 'hostqa':

# hostqa.sls:
test:
  setting1: A2

The commands salt hostqa saltutil.refresh_pillar and salt hostqa pillar.data will then show that the values A2 and B2 as they have all been merged together.

As this works without specifying environments, I suggest not to use environments here.

Override some local settings outside of Git

To override some of your settings locally, you can add another external pillar. One of the most simple ones is cmd_yaml that will run a command (here: cat) and merge the output with the current pillar:

ext_pillar:
  - git:
    - master https://gitserver/git-pillar.git:
      - env: base
  - cmd_yaml: cat /srv/salt/local_override.sls

All external pillars are executed in the order listed in the configuration file.