Create Gitlab CI job through for loop

Question

I have 100 folders in my repo. I will need to create one job for each folder under one stage.

How can I create multiple job through a for loop in gitlab-ci.yml to be displayed under one stage?

Answer 1

There is no "loop" construct in YAML or GitLab CI definitions, but there's a few ways you might approach this problem. I'll describe 3 possible approaches in order of practicality.

1. Dynamic child pipelines

You can use a job to create a dynamic child pipeline yaml programmatically. This is probably the approach with the lowest maintenance and fewest restrictions, if you're willing to script the child pipeline creation, as documented in the link.

generate-config:
  script:
    # use a script to generate a YAML containing the 100 jobs you want
    - generate-ci-config > generated-config.yml
  artifacts:
    paths:
      - generated-config.yml

child-pipeline:
  needs: [generate-config]
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-config

You would need to implement the script, but in Python it might look something like this:

#!/usr/bin/env python3

import os

JOB_TEMPLATE = """\
"{name}_job":
  variables:
    DIRNAME: "{name}"
  script:
    - make "$DIRNAME"

"""

generated_yaml = ''

for name in os.path.listdir():
    if name.startswith('.') or not os.path.isdir(name):
        continue  # or use whatever condition makes sense for you
    job_definition = JOB_TEMPLATE.format(name=name)
    generated_yaml += job_definition

print(generated_yaml)

2. Wildstar includes

With this approach, you would create a yaml file in in each directory, for example like folder1/job.yml, folder2/job.yml, etc. and commit this to the repo. Each file should contain a uniquely named job definition. Then in your .gitlab-ci.yml use an include with wildstar glob to include all the yaml files.

include:
  - local: "*/job.yml"

This can be approached similar to the dynamic child pipeline approach discussed above, except perhaps you might script the generation of the */job.yml files programmatically and commit them instead. This would also avoid the child pipeline view, if you find that desirable.

Note that there is a default limit of 150 includes -- self-managed instances can increase this limit. There are also size limits, but this is unlikely to be an issue.

3. Parallel Matrix

Use parallel:matrix: to create 100 jobs from a single definition. However, note that you'll have to write each of the 100 directory names into the YAML in this approach. This is probably the least flexible approach.

Note: there is a limit of 200 permutations allowed with this method.

myjob:
  script: # or whatever
    - make $DIRECTORY_NAME
  parallel:
    matrix:
      # edit to include all your directories...
      - DIRECTORY_NAME: [folder1, folder2, folder3, etc]