Gitlab CI: Set dynamic variables

Question

For a gitlab CI I'm defining some variables like this:

variables:
  PROD: project_package
  STAGE: project_package_stage
  PACKAGE_PATH: /opt/project/build/package
  BUILD_PATH: /opt/project/build/package/bundle
  CONTAINER_IMAGE: registry.example.com/project/package:e2e

I would like to set those variables a bit more dynamically, as there are mainly only two parts: project and package. Everything else depends on those values, that means I have to change only two values to get all other variables.

So I would expect something like

variables:
  PROJECT: project
  PACKAGE: package
  PROD: $PROJECT_$PACKAGE
  STAGE: $PROD_stage
  PACKAGE_PATH: /opt/$PROJECT/build/$PACKAGE
  BUILD_PATH: /opt/$PROJECT/build/$PACKAGE/bundle
  CONTAINER_IMAGE: registry.example.com/$PROJECT/$PACKAGE:e2e

But it looks like, that the way doing this is wrong...

Answer 1

I don't know where your expectation comes from, but it is trivial to check there is no special meaning for $, _, '/' nor : if not followed by a space in YAML. There might be in gitlab, but I doubt strongly that there is in the way you expect.

To formalize your expectation, you assume that any key (from the same mapping) preceded by a $ and terminated by the end of the scalar, by _ or by / is going to be "expanded" to that key's value. The _ has to be such terminator otherwise $PROJECT_$PACKAGE would not expand correctly.

Now consider adding a key-value pair:

 BREAKING_TEST: $PACKAGE_PATH

is this supposed to expand to:

 BREAKING_TEST: /opt/project/build/package/bundle

or follow the rule you implied that _ is a terminator and just expand to:

 BREAKING_TEST: project_PATH

To prevent this kind of ambiguity programs like bash use quoting around variable names to be expanded ( "$PROJECT"_PATH vs. $PROJECT_PATH), but the more sane, and modern, solution is to use clamping begin and end characters (e.g. { and }, $% and %, ) with some special rule to use the clamping character as normal text.

So this is not going to work as you indicated as indeed you do something wrong.

It is not to hard to pre-process a YAML file, and it can be done with e.g. Python (but watch out that { has special meaning in YAML), possible with the help of jinja2: load the variables, and then expand the original text using the variables until replacements can no longer be made.

But it all starts with choosing the delimiters intelligently. Also keep in mind that although your "variables" seem to be ordered in the YAML text, there is no such guarantee when the are constructed as dict/hash/mapping in your program.

You could e.g. use << and >>:

variables:
  PROJECT: project
  PACKAGE: package
  PROD: <<PROJECT>>_<<PACKAGE>>
  STAGE: <<PROD>>_stage
  PACKAGE_PATH: /opt/<<PROJECT>>/build/<<PACKAGE>>
  BUILD_PATH: /opt/<<PROJECT>>/build/<<PACKAGE>>/bundle
  CONTAINER_IMAGE: registry.example.com/<<PROJECT>>/<<PACKAGE>>:e2

which, with the following program (that doesn't deal with escaping << to keep its normal meaning) generates your original, expanded, YAML exactly.

import sys
from ruamel import yaml


def expand(s, d):
    max_recursion = 100
    while '<<' in s:
        res = ''
        max_recursion -= 1
        if max_recursion < 0:
            raise NotImplementedError('max recursion exceeded')
        for idx, chunk in enumerate(s.split('<<')):
            if idx == 0:
                res += chunk  # first chunk is before <<, just append
                continue
            try:
                var, rest = chunk.split('>>', 1)
            except ValueError:
                raise NotImplementedError('delimiters have to balance "{}"'.format(chunk))
            if var not in d:
                res += '<<' + chunk
            else:
                res += d[var] + rest
        s = res
    return s


with open('template.yaml') as fp:
    yaml_str = fp.read()
variables = yaml.safe_load(yaml_str)['variables']
data = yaml.round_trip_load(expand(yaml_str, variables))
yaml.round_trip_dump(data, sys.stdout, indent=2)