Github Actions create matrix of multiple sequential jobs

Question

I am trying to create a workflow like the following, where the matrix doesn't consist of just one job but multiple jobs, for each env we want to build, test, and deploy.

If a step in an env fails, the further steps for the env shouldn't run.

Matrix is like ["Env A", "Env B", ... , "Env n"]

• Want to avoid repeating all the jobs for each env.
• Can not use single matrix job for build, then single matrix job for test etc because dependence b/w steps in an env will not be maintained.

Any other way to do it without repeating code?

Answer 1

You could use reusable workflows with different inputs / secrets for each env. It will work with matrix as well.

Example:

Using a first workflow with 3 jobs:

• The first one to set the matrix.
• The second one to execute the reusable by environment in sequence according to matrix input.
• The third one to finalize after performing operations for each env.

on:
  push:

jobs:
  common-job-1:
    runs-on: ubuntu-latest
    outputs: 
      environments: ${{ steps.environments.outputs.environments }}
    steps:
      - name: Build Environments Array
        id: environments
        run: |
          myArray=()
          myArray+=("env1")
          myArray+=("env2")
          myArray+=("env3")
          myArray=$(jq --compact-output --null-input '$ARGS.positional' --args -- "${myArray[@]}")
          echo "Updated environments list: $myArray"
          echo "environments=$myArray" >> $GITHUB_OUTPUT
        shell: bash

  reusable:
    needs: [common-job-1]
    strategy:
      matrix:
        environment: ${{ fromJSON(needs.common-job-1.outputs.environments) }}
      fail-fast: true
      max-parallel: 1
    # environment:
    #   name: ${{ matrix.environment }}
    uses: OWNER/REPO/.github/workflows/reusable.yml@main
    with:
      stage: ${{ matrix.environment }}

  common-job-2:
    needs: [common-job-1, reusable]
    runs-on: ubuntu-latest
    steps:
      - name: Do something
        run: echo "Do something"

Note: workflow file can be found here

With a reusable workflow like this:

on:
  workflow_call:
    inputs:
      stage:
        required: true
        type: string

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Build in ${{ inputs.stage }}
        run: |
          echo "Build in ${{ inputs.stage }}"

  test:
    runs-on: ubuntu-latest
    steps:
      - name: Test in ${{ inputs.stage }}
        run: |
          echo "Test in ${{ inputs.stage }}"

  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy in ${{ inputs.stage }}
        run: |
          echo "Deploy in ${{ inputs.stage }}"

Note: workflow file can be found here

When using max-parallel: 1 in the matrix strategy, the order of the list used as matrix input is respected and matrix runs will follow the same order (reference for more details).

Note that if you remove the max-parallel: 1 field, the workflow will be executed faster (as all jobs will be run in parallel) and will still execute the build, test and deploy operation in each environment.

The thing is that depending on your context, you may want to run the deploy in an environment before running it in the others.

---

Here is the output of a workflow run following the implementation above (with other job names) using max-parallel: 1 in the matrix strategy.

Observation: The Github workflow run image UI doesn't help, but you can see on the left side that each job have been executed as you expected in your question.

The last job waited for all reusable jobs to run all 3 operations (build, test and deploy).

Here is the example without using max-parallel: 1 in the matrix strategy. As you can see, it is a lot faster, however the matrix execution order may not be respected.