Variable expansion of trigger branch property prevents downstream pipeline from being created

Question

A branch job in which the branch property of the trigger property is using a variable will always fail with reason: downstream pipeline can not be created.

Steps to reproduce

1. Set up a downstream pipeline with a trigger property as you would normally.
2. Add a branch property to the trigger property. Write the name of an existing branch on the downstream repository, like master/main or the name of a feature branch.
3. Run the pipeline and observe that the downstream pipeline is successfully created.
4. Now change the branch property to use a variable instead, like branch: $CI_TARGET_BRANCH.
5. Manually run the CI pipeline with that, setting variable through the GitLab GUI.
6. The job will instantly fail with reason: downstream pipeline can not be created.

Code example

The goal is to create a GitLab CI config that runs the pipeline of a specified downstream branch. The bug occurs when attempting to do it with a variable.

This works, creating a downstream pipeline like normal. But the branch name is hardcoded:

stages:
  - deploy

deploy:
  variables:
    environment: dev
  stage: deploy
  trigger:
    project: group/project
    branch: foo
    strategy: depend

This does not work; although TARGET_BRANCH is set successfully, the job fails because the downstream pipeline can not be created:

stages:
  - removeme
  - deploy

before_script:

  - if [ -z "$TARGET_BRANCH" ]; then TARGET_BRANCH="main"; fi
  - echo $TARGET_BRANCH

test_variable:
  stage: removeme
  script:
    - echo $TARGET_BRANCH

deploy:
  variables:
    environment: dev
  stage: deploy
  trigger:
    project: group/project
    branch: $TARGET_BRANCH
    strategy: depend

If you know what I'm doing wrong, or you have something that does work with variable expansion of the branch property, please share it (along with your GitLab version). Alternate solutions are also welcome, but this one seems like it should work.

GitLab Version on which bug occurs

Self-hosted GitLab Community Edition 12.10.7

What is the current bug behavior?

The job always fails for reason: downstream pipeline can not be created.

What is the expected correct behavior?

The branch property should be set to the value of the variable and the downstream pipeline should be created as normal, just as if you simply hardcoded/typed the name of the branch.

More details

• The ability to use variable expansion in the trigger branch property was added in v12.4, and it's explicitly mentioned in the docs.
• I searched for other .gitlab-ci.yml / GitLab config files. Every single one that attempted to use variable expansion in the branch property had it commented out, saying it was bugged for an unknown reason (example.
  • I haven't been able to find a repository in which someone claimed to have a working variable expansion for the branch property of the trigger property.
• Unfortunately, the alternate solutions are either (a) hardcoding every downstream branch name into the GitLab CI config of the upstream project, or (b) not being able to test changes to the downstream GitLab CI config without first committing them to master/main, or having to use only/except.

---

TL;DR: How to use the value of a variable for the branch property of a bridge job? My current solution makes it so the job fails and the downstream pipeline isn't created.

Answer 1

this is a 'works as designed', and gitlab will improve in upcoming releases.

trigger job will pretty weak b/c it is not a full job that runs on a runner. Therefore most of the trigger configuration needs to be hardcoded.

I use direct API calls to trigger downstream jobs passing the CI_JOB_TOKEN which links the upstream job to downstream as the trigger does

API calls give you full control

curl -X POST \
          -s \
          -F token=${CI_JOB_TOKEN} \
          -F "ref=${REF_NAME}" \
          -F "variables[STAGE]=${STAGE}" \
          "${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/trigger/pipeline"

now this will not wait and monitor for when the job is done so you will need to code for that if you need to wait for the downstream job to finish,

Moreover, CI_JOB_TOKEN cannot be used to get the status of the downstream job, so you will another token for that.

- |

DOWNSTREAM_RESULTS=$( curl --silent  -X POST \
      -F token=${CI_JOB_TOKEN} \
      -F "ref=${DOWNSTREAM_PROJECT_REF}" \
      -F "variables[STAGE]=${STAGE}" \
      -F "variables[SLS_PACKAGE_PATH]=.serverless-${STAGE}" \
      -F "variables[INVOKE_SLS_TESTS]=false" \
      -F "variables[UPSTREAM_PROJECT_REF]=${CI_COMMIT_REF_NAME}" \
      -F "variables[INSTALL_SLS_PLUGINS]=${INSTALL_SLS_PLUGINS}" \
      -F "variables[PROJECT_ID]=${CI_PROJECT_ID}" \
      -F "variables[PROJECT_JOB_NAME]=${PROJECT_JOB_NAME}" \
      -F "variables[PROJECT_JOB_ID]=${PROJECT_JOB_ID}" \
      "${CI_SERVER_URL}/api/v4/projects/${DOWNSTREAM_PROJECT_ID}/trigger/pipeline" )
      echo ${DOWNSTREAM_RESULTS} | jq .
      DOWNSTREAM_PIPELINE_ID=$( echo ${DOWNSTREAM_RESULTS} | jq -r .id )
      echo "Monitoring Downstream pipeline ${DOWNSTREAM_PIPELINE_ID} status..."
      DOWNSTREAM_STATUS='running'
      COUNT=0
      PIPELINE_API_URL="${CI_SERVER_URL}/api/v4/projects/${DOWNSTREAM_PROJECT_ID}/pipelines/${DOWNSTREAM_PIPELINE_ID}"
      echo "Pipeline api endpoint => ${PIPELINE_API_URL}"
      while [ ${DOWNSTREAM_STATUS} == "running" ]
      do
         if [ $COUNT -eq 0  ]
        then
          echo "Starting loop"
        fi
        if [ ${COUNT} -ge 350 ]
        then
          echo 'TIMEOUT!'
          DOWNSTREAM_STATUS="TIMEOUT"
          break
        elif [ $(( ${COUNT}  % 60 )) -eq 0 ]
        then
          echo "Downstream pipeline status => ${DOWNSTREAM_STATUS}"
          echo "Count => ${COUNT}"
          sleep 10 
        else 
          sleep 10 
        fi
        DOWNSTREAM_CALL=$( curl --silent --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" ${PIPELINE_API_URL} )
        if [ $COUNT -eq 0  ]
        then
          echo ${DOWNSTREAM_CALL} | jq .
        fi
        DOWNSTREAM_STATUS=$( echo ${DOWNSTREAM_CALL} | jq -r .status )
        COUNT=$(( ${COUNT} + 1 ))

      done
      #pipeline status is running, failed, success, manual
      echo "PIPELINE STATUS => ${DOWNSTREAM_STATUS}"
      if [ ${DOWNSTREAM_STATUS} != "success" ]
      then
        exit 2
      fi