Ansible - with_dict: dictionary - How to use variables defined in each dictionary which depends upon others

Question

Environment is: Ansible 1.9.2, CentOS 6.5

I have created a role to download JAVA (.tar.gz) artifact files for 3 different JAVA versions from Artifactory. I'm trying to use Ansible's with_dict feature (instead of using with_items).

Created the following files:

$ cat roles/java/defaults/main.yml

---
java_versions:
  java7_60:
    version: 1.7.60
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
    dist_file: "jdk-{{ version }}-{{ classifier }}-{{ ext }}"
#    dist_file: "jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    dist_url: "{{ artifactory_url }}/{{ group_path }}/{{ version }}/{{ dist_file }}"
#    dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"

  java7_67:
    version: 1.7.67
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
    dist_file: "jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"
  java8_45:
    version: 1.8.45
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
    dist_file: "jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"

How can I set or use dist_file or dist_url variables which depends upon other variables defined in the same KEY (lets say in KEY java7_60)?

Right now, when I'm trying either the current dist_file or dist_url variables OR the commented out lines way of setting them (i.e. using item.value.), it's not setting the value of these 2 variables as desired i.e. depending upon other variables version, group_path, classifier, ext and artifactory_url (which is defined in another common role's defaults/main.yml file)).

I saw that for using with_dict: within a playbook/task, I have to use {{ item.value.variable_name }} but how can I define a variable which is dependent upon others within the same KEY section of a dictionary.

The error message I'm getting while using the above dictionary in the following task is:

$ cat roles/java/tasks/main.yml:

- name: Download Java/JDK Versions
  command: wget -q "{{ item.value.dist_url }}"
    chdir="{{ common_download_dir }}"
    creates="{{ common_download_dir }}/{{ item.value.dist_file }}"
  with_dict: "{{ java_versions }}"
  become_user: "{{ build_user }}"

Error message with using dist_file / dist_url (with the current setup in roles/java/defaults/main.yml):

TASK: [java | Download Java/JDK Versions] *************************************
failed: [server01.poc.jenkins] => (item={'key': 'java7_60', 'value': {'dist_file': u'jdk-{# version #}-{# classifier #}-{# ext #}', 'ext': 'tar.gz', 'version': '1.7.60', 'dist_url': u'{# artifactory_ur #}/{# group_path #}/{# version #}/{# dist_file #}', 'group_path': 'com/oracle/jdk', 'classifier': 'linux-x64'}}) => {"changed": true, "cmd": ["wget", "-q", "{# artifactory_url #}/{# group_path #}/{# version }/{# dist_file #}"], "delta": "0:00:00.006081", "end": "2015-11-23 12:50:18.383728", "item": {"key": "java7_60", "value": {"classifier": "linux-x64", "dist_file": "jdk-{# version #}-{# classifier #}-{# ext #}, "dist_url": "{# artifactory_url #}/{# group_path #}/{# version #}/{# dist_file #}", "ext": "tar.gz", "group_path": "com/oracle/jdk", "version": "1.7.60"}}, "rc": 4, "start": "2015-11-23 12:50:18.377647", "wrnings": ["Consider using get_url module rather than running wget"]}

Error message with using dist_file / dist_url (with the lines which are currently commented out in roles/java/defaults/main.yml):

TASK: [java | Download Java/JDK Versions] *************************************
failed: [server01.poc.jenkins] => (item={'key': 'java7_60', 'value': {'dist_file': u'jdk-{#item.value.version #}-{# item.value.classifier #}-{# item.value.ext #}', 'ext': 'tar.gz', 'version': '1.7.60' , 'dist_url': u'{# artifactory_url #}/{# item.value.group_path #}/{# item.value.version #}/{# dist_file #}', 'group_path': 'com/oracle/jdk', 'classifier': 'linux-x64'}}) => {"changed": true, "cmd": ["wget",  "-q", "{# artifactory_url #}/{# item.value.group_path #}/{# item.value.version #}/{# dist_file #}"], "delta": "0:00:00.005900", "end": "2015-11-23 12:36:24.131327", "item": {"key": "java7_60", "value": {"cla ssifier": "linux-x64", "dist_file": "jdk-{#item.value.version #}-{# item.value.classifier #}-{# item.value.ext #}", "dist_url": "{# artifactory_url #}/{# item.value.group_path #}/{# item.value.version #}/{#  dist_file #}", "ext": "tar.gz", "group_path": "com/oracle/jdk", "version": "1.7.60"}}, "rc": 4, "start": "2015-11-23 12:36:24.125427", "warnings": ["Consider using get_url module rather than running wget"]}

Answer 1

I was curious about this so I did a bit of digging, and in doing so I came across this similar answer. That's only part of the solution in your case though.

It appears that Ansible won't let you reference a variable from its own definition, which I guess makes sense since it's not fully defined. So something like this won't work, and will in fact throw a somewhat confusing error when the variable is referenced:

---
myvar:
    param1: foo
    param2: "{{ myvar['foo'] }} bar"

It also appears, from your own example, that Ansible won't let you use item constructs in variables to reference other complex variables. This sort of makes sense to me since it seems Ansible resolves jinja2 constructs in variables at the time the variable is defined, not at runtime when the variable is referenced.

So although this isn't exactly what you were looking for, I think if you break apart your variable into two parts you can get this to work by doing something along these lines:

---
artifactory_url: "http://path.to.jarfile"
java_versions:
  java7_60:
    version: 1.7.60
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz

java_downloads:
  java7_60:
    dist_url: "{{ artifactory_url }}/{{ java_versions['java7_60']['group_path'] }}/{{ java_versions['java7_60']['version'] }}/jdk-{{ java_versions['java7_60']['version'] }}-{{ java_versions['java7_60']['classifier'] }}.{{ java_versions['java7_60']['ext'] }}"

When you debug java_downloads this way you get the full URL you're looking for:

TASK: [debug var=item] ********************************************************
ok: [localhost] => (item={'key': 'java7_60', 'value': {'dist_url': u'http://path.to.jarfile/com/oracle/jdk/1.7.60/jdk-1.7.60-linux-x64.tar.gz'}}) => {
    "item": {
        "key": "java7_60",
        "value": {
            "dist_url": "http://path.to.jarfile/com/oracle/jdk/1.7.60/jdk-1.7.60-linux-x64.tar.gz"
        }
    },
    "var": {
        "item": {
            "key": "java7_60",
            "value": {
                "dist_url": "http://path.to.jarfile/com/oracle/jdk/1.7.60/jdk-1.7.60-linux-x64.tar.gz"
            }
        }
    }
} 

Answer 2

I believe the best way to do what you want (since I don't believe loop variables can reference themselves) would be to have your task be:

- name: Download Java/JDK Versions
  command: wget -q "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    chdir="{{ common_download_dir }}"
    creates="{{ common_download_dir }}/jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
  with_dict: "{{ java_versions }}"
  become_user: "{{ build_user }}"

essentially manually inserting the variables into your task.

It's not pretty if you need to use the variables for multiple tasks, but in ansible 1.x I don't think there's a way to make it pretty. Ansible 2.0 has blocks, with which you would be able to loop multiple tasks together over a dict, and you would be able to define variables for all tasks in that block to use.