How to join two Salt pillar files and merge data?

Question

Is there any way to join two pillar files?

I have a users pillar. It's something like:

users:
  joe:
    sudouser: True
  jack:
    sudouser: False

Now I need different set of users for certain servers (ie. add some users to one server). So I create new pillar file:

users:
  new_user:
    sudouser: True

And assign this topfile to the server. But because the key is the same it would overwrite the first one. If I change it I would need to update the state file (which I really don't want). How should I approach this problem? Is there any way to tell salt to "merge" the files?

Answer 1

It is possible at least according to the latest Salt documentation about pillar (as of 5188d6c) which states:

With some care, the pillar namespace can merge content from multiple pillar files under a single key, so long as conflicts are avoided ...

I tested it under Salt Helium (2014.7.0) and it's working as expected.

---

Your Example

Pillar file user_set_a.sls:

users:
  joe:
    sudouser: True
  jack:
    sudouser: False

Pillar file user_set_b.sls:

users:
  new_user:
    sudouser: True

Run pillar.items to confirm that all users are merged under the same users key:

salt-call pillar.items
...
    users:
    ----------
    jack:
        ----------
        sudouser:
            False
    joe:
        ----------
        sudouser:
            True
    new_user:
        ----------
        sudouser:
            True
...

---

See also:

• Example to include pillar files under sub-keys: https://serverfault.com/a/591501/134406

Answer 2

Short answer: you can't merge pillar data in this way.

Long answer: the pillar doesn't support the extend keyword the same way the state tree does, though there is some conversation on salt issue #3991. Unfortunately, there doesn't seem to be any real momentum with this at the moment and I'm not aware of any plans for this to be included in Helium.

Realistically, you'd be better off ensuring that your pillar data is distinct on a per-minion basis, and then you won't need to worry about collisions. You could optionally do something with YAML anchors and references, e.g.

# common/base users.sls
base_users: &base_users
    user1:
       foo: bar
    user2:
       baz: bat

# minion1.sls
{% include 'common/base_users.sls' %}

users:
    <<: *base_users
    user3:
        qux: quux

# minion2.sls
{% include 'common/base_users.sls' %}

users:
    <<: *base_users
    user4:
        corge: grault

Another potential (hacky) option is to use an external pillar module and do some sort of glob matching on pillar keys provided to the module, so you could basically have keys like merge-thing-abc123 and merge-thing-def456, using the merge prefix to group by thing and combine the data. I wouldn't really recommend this as it's a pretty blatant antipattern WRT pillar data (not to mention difficult to maintain).

For what it's worth, this is something that also frustrates me occasionally, but I end up deciding that some minimal data duplication is better than coming up with a workaround. Using the YAML references, this could potentially be a more agreeable option since technically you don't need to duplicate data, and is more easily maintainable. Granted, you end up polluting the pillar with extra unused keys (e.g. base_users), but in this particular case I'd consider that acceptable.

Hope this helps!

Edit: I may have spoke too soon; it looks as though includes are parsed prior to being injected into the including file, so anchors/references wouldn't work in that case. Looking into it, will update.

Edit 2: Just occurred to me that since both state and pillar files are essentially Python modules, they can be included with Jinja vs using pillar's include. So, instead of

include:
    - common.base_users

you can do

{% include 'common/base_users.sls' %}

and then proceed to reference any anchors defined in the included document. Updated the original answer to illustrate this (verified to work).