Extract and manipulate dict data to check certificates

Question

I struggle on a regular basis with data manipulation in Ansible. I'm not very familiar with Python and dict objects. I found an example that sums up a lot of my misunderstandings.

I would like to verify a list of certificates. In found an example for a single domain in the documentation, I'm just trying to loop over several domain names.

Certs are stored in a folder:

certs/
├── domain.com
│   ├── domain.com.pem
│   └── domain.com.key
└── domain.org
    ├── domain.org.key
    └── domain.org.pem

My playbook is as follow:

---
- name: "check certs"
  hosts: localhost
  gather_facts: no
  vars:
    domain_names:
      - domain.com
      - domain.org
    certs_folder: certs
  tasks: 
    - name: Get certificate information
      community.crypto.x509_certificate_info:
        path: "{{ certs_folder }}/{{ item }}/{{ item }}.pem"
        # for valid_at, invalid_at and valid_in
      register: result_certs
      loop: "{{ domain_names }}"
      failed_when: 0

    - name: Get private key information
      community.crypto.openssl_privatekey_info:
        path: "{{ certs_folder }}/{{ item }}/{{ item }}.key"
      register: result_privatekey
      loop: "{{ domain_names }}"
      failed_when: 0

    - name: Check cert and key match << DOES NOT WORK >>>
      assert: 
        that:
          - result_certs[ {{ item }} ].public_key == result_privatekey[ {{ item }} ].public_key
          # - ... other checks ...
          - not result[ {{ item }} ].expired
      loop: "{{ domain_names }}"

So I get two variables result_certs and result_privatekey, each has a element result which is , if I understand correctly, an array of dicts:

"result_certs": {
        "changed": false,
        "msg": "All items completed",
        "results": [
            {
                "expired": false,
                "item": "domain.org",
                "public_key": "<<PUBLIC KEY>>"
            },
            {
                "expired": false,
                "item": "domain.com",
                "public_key": "<<PUBLIC KEY>>"
            }
        ],
        "skipped": false
    }

"result_privatekey": {
    "changed": false,
    "msg": "All items completed",
    "results": [
    {
        "item": "domain.org",
        "public_key": "<< PUBLIC KEY >>"
    },
    {
        "item": "domain.com",
        "public_key": "<< PUBLIC KEY >>"
    }
    ],
    "skipped": false
}

How can I refer to each of the dicts elements like result_privatekey.results[the dict where item ='domain.org'].public_key in the assert task?

I feel like I'm missing something, or a documentation page to make it clear to me. I noticed that I particularly struggle with arrays of dicts, and I run into those objects quite often...

I found those resources useful, but not sufficient to get this job done:

https://jmespath.org/tutorial.html
https://jinja.palletsprojects.com/en/3.0.x/templates/

EDIT: map and selectattr are the filters required to solve this problem, although the documentation (including the official ansible doc) is not that clear to me... This is very useful to get many tutorial examples on those two filters if one is struggling as I do.

Vladimir Botka · Accepted Answer

^{Given the simplified data for testing
result_certs:
changed: false
msg: All items completed
results:
- expired: false
item: domain.org
public_key: <<PUBLIC KEY domain.org>>
- expired: false
item: domain.com
public_key: <<PUBLIC KEY domain.com>>
skipped: false

result_privatekey:
changed: false
msg: All items completed
results:
- item: domain.org
public_key: <<PUBLIC KEY domain.org>>
- item: domain.com
public_key: <<PUBLIC KEY domain.com>>
skipped: false

Declare the list of the domains
domains: "{{ result_certs.results|
map(attribute='item')|list }}"

gives
domains:
- domain.org
- domain.com}

Q: "How can I refer to each dictionary element?"

A: select the item(s) and map the attribute

    - debug:
        var: pk
      loop: "{{ domains }}"
      vars:
        pk: "{{ result_privatekey.results|
                selectattr('item', '==', item)|
                map(attribute='public_key')|list }}"

gives

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) => 
  ansible_loop_var: item
  item: domain.org
  pk:
  - <<PUBLIC KEY domain.org>>
ok: [localhost] => (item=domain.com) => 
  ansible_loop_var: item
  item: domain.com
  pk:
  - <<PUBLIC KEY domain.com>>

In the same way, you can compare the keys in the loop

    - debug:
        msg: "{{ pk1 }} == {{ pk2 }}: {{ pk1 == pk2 }}"
      loop: "{{ domains }}"
      vars:
        pk1: "{{ result_privatekey.results|
                 selectattr('item', '==', item)|
                 map(attribute='public_key')|first }}"
        pk2: "{{ result_certs.results|
                 selectattr('item', '==', item)|
                 map(attribute='public_key')|first }}"

gives

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) => 
  msg: '<<PUBLIC KEY domain.org>> == <<PUBLIC KEY domain.org>>: True'
ok: [localhost] => (item=domain.com) => 
  msg: '<<PUBLIC KEY domain.com>> == <<PUBLIC KEY domain.com>>: True'

Q: "Extract and manipulate dict data to check certificates."

A: There are many options:

For example, create a unique list of all public keys

  pkeys: "{{ result_certs.results|
             zip(result_privatekey.results)|
             map('map', attribute='public_key')|
             map('unique')|flatten }}"

gives

  pkeys:
  - <<PUBLIC KEY domain.org>>
  - <<PUBLIC KEY domain.com>>

To find the redundant keys compare the lists pkeys and domains. Compare the lengths to briefly find out if there are any

  pkeys|length == domains|length

To find the expired domains declare variables

  expired: "{{ result_certs.results|
               map(attribute='expired')|list }}"
  expired_domains: "{{ result_certs.results|
                       selectattr('expired')|
                       map(attribute='item')|list }}"

give

  expired:
  - false
  - false

  expired_domains: []

Then the assert task should look like

    - assert:
        that:
          - expired is not any
          - pkeys|length == domains|length

^{Example of a complete playbook for testing
- hosts: localhost

vars:

result_certs:
changed: false
msg: All items completed
results:
- expired: false
item: domain.org
public_key: <<PUBLIC KEY domain.org>>
- expired: false
item: domain.com
public_key: <<PUBLIC KEY domain.com>>
skipped: false

result_privatekey:
changed: false
msg: All items completed
results:
- item: domain.org
public_key: <<PUBLIC KEY domain.org>>
- item: domain.com
public_key: <<PUBLIC KEY domain.com>>
skipped: false

domains: "{{ result_certs.results|
map(attribute='item')|list }}"
pkeys: "{{ result_certs.results|
zip(result_privatekey.results)|
map('map', attribute='public_key')|
map('unique')|flatten }}"
expired: "{{ result_certs.results|
map(attribute='expired')|list }}"
expired_domains: "{{ result_certs.results|
selectattr('expired')|
map(attribute='item')|list }}"

tasks:

- debug:
var: domains

- debug:
var: pkeys

# How can I refer to each of the dicts elements?
- debug:
var: pk
loop: "{{ domains }}"
vars:
pk: "{{ result_privatekey.results|
selectattr('item', '==', item)|
map(attribute='public_key')|list }}"

# How can I compare private keys?
- debug:
msg: "{{ pk1 }} == {{ pk2 }}: {{ pk1 == pk2 }}"
loop: "{{ domains }}"
vars:
pk1: "{{ result_privatekey.results|
selectattr('item', '==', item)|
map(attribute='public_key')|first }}"
pk2: "{{ result_certs.results|
selectattr('item', '==', item)|
map(attribute='public_key')|first }}"
- debug:
var: expired
- debug:
var: expired_domains

- assert:
that:
- expired is not any
- pkeys|length == domains|length

gives
PLAY [localhost] *****************************************************************************

TASK [debug] *********************************************************************************
ok: [localhost] =>
domains:
- domain.org
- domain.com

TASK [debug] *********************************************************************************
ok: [localhost] =>
pkeys:
- <<PUBLIC KEY domain.org>>
- <<PUBLIC KEY domain.com>>

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) =>
ansible_loop_var: item
item: domain.org
pk:
- <<PUBLIC KEY domain.org>>
ok: [localhost] => (item=domain.com) =>
ansible_loop_var: item
item: domain.com
pk:
- <<PUBLIC KEY domain.com>>

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) =>
msg: '<<PUBLIC KEY domain.org>> == <<PUBLIC KEY domain.org>>: True'
ok: [localhost] => (item=domain.com) =>
msg: '<<PUBLIC KEY domain.com>> == <<PUBLIC KEY domain.com>>: True'

TASK [debug] *********************************************************************************
ok: [localhost] =>
expired:
- false
- false

TASK [debug] *********************************************************************************
ok: [localhost] =>
expired_domains: []

TASK [assert] ********************************************************************************
ok: [localhost] => changed=false
msg: All assertions passed

PLAY RECAP ***********************************************************************************
localhost: ok=7 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0}

The next option is to create a structure for this purpose. For example, put into the lists the attributes which might have different values (i.e. public_key in this case). Merge the dictionaries by the domain and append unique attributes. Declare the variables

  l1: "{{ result_certs.results|json_query('[].{item: item,
                                               expired: expired,
                                               pk: [public_key]}') }}"
  l2: "{{ result_privatekey.results|json_query('[].{item: item,
                                                    pk: [public_key]}') }}"
  lm: "{{ [l1, l2]|community.general.lists_mergeby('item', 
                                                    list_merge='append_rp') }}"

gives

  lm:
  - expired: false
    item: domain.com
    pk:
    - <<PUBLIC KEY domain.com>>
  - expired: false
    item: domain.org
    pk:
    - <<PUBLIC KEY domain.org>>

Use this structure to compare any attributes. For example, declare

  exprd: "{{ lm|map(attribute='expired')|list }}"
  pkeys: "{{ lm|map(attribute='pk')|
                map('length')|sum }}"

gives

  exprd:
  - false
  - false

  pkeys: '2'

Then, use it in the conditions

    - assert:
        that:
          - exprd is not any
          - pkeys|int == lm|length

^{Example of a complete playbook for testing
- hosts: localhost

vars:

result_certs:
changed: false
msg: All items completed
results:
- expired: false
item: domain.org
public_key: <<PUBLIC KEY domain.org>>
- expired: false
item: domain.com
public_key: <<PUBLIC KEY domain.com>>
skipped: false

result_privatekey:
changed: false
msg: All items completed
results:
- item: domain.org
public_key: <<PUBLIC KEY domain.org>>
- item: domain.com
public_key: <<PUBLIC KEY domain.com>>
skipped: false

l1: "{{ result_certs.results|json_query('[].{item: item,
expired: expired,
pk: [public_key]}') }}"
l2: "{{ result_privatekey.results|json_query('[].{item: item,
pk: [public_key]}') }}"
lm: "{{ [l1, l2]|
community.general.lists_mergeby('item',
list_merge='append_rp') }}"

exprd: "{{ lm|map(attribute='expired')|list }}"
pkeys: "{{ lm|map(attribute='pk')|
map('length')|sum }}"
tasks:

- debug:
var: l1
- debug:
var: l2
- debug:
var: lm
- debug:
var: exprd
- debug:
var: pkeys

- assert:
that:
- exprd is not any
- pkeys|int == lm|length}

In addition to the structure created in option 2) create dictionaries to test the attributes selectively. For example,

  domain_exprd: "{{ lm|items2dict(key_name='item', value_name='expired') }}"
  domain_pkeys: "{{ lm|items2dict(key_name='item', value_name='pk') }}"

gives

  domain_exprd:
    domain.com: false
    domain.org: false

  domain_pkeys:
    domain.com:
    - <<PUBLIC KEY domain.com>>
    domain.org:
    - <<PUBLIC KEY domain.org>>

β.εηοιτ.βε · Answer

From what you are showing — but you might have more conditions that needs it — I wouldn't loop on the domain_names in your assertion task, I would rather loop on result_certs.

From there on, you can select the corresponding private key thanks to the selectattr filter.

So, your assertion would become:

- assert: 
    that:
      - >-
          item.public_key == (
            result_privatekey.results 
              | selectattr('item', '==', item.item) 
              | first
          ).public_key
      # - ... other checks ...
      - not item.expired
  loop: "{{ result_certs.results }}"

Extract and manipulate dict data to check certificates

Tags:

python

jquery

dictionary

ansible

jeanlabab

2 Answers

Vladimir Botka

β.εηοιτ.βε

Recent Activity

Donate For Us

Extract and manipulate dict data to check certificates

Tags:

python

jquery

dictionary

ansible

jeanlabab

2 Answers

Vladimir Botka

β.εηοιτ.βε

Related questions

Recent Activity

Donate For Us