Github Actions: How use strategy/matrix with script

Question

I have workflow that needs to have a loop for the steps, which is perfect with strategy/matrix.

The only problem is that strategy/matrix needs to be set by a constant.

Is it possible to use strategy matrix with a output of a script?

name: tests
on: [push]

jobs:
  test:
    runs-on: ${{ ubuntu-latest }}
    strategy:
      fail-fast: false
      matrix:
        versions: $(./script.py)

    steps:
    - uses: actions/checkout@v2
 .......

Answer 1

You can generate matrix in JSON in one job and set it to the second job.

GitHub added this feature in April: https://github.blog/changelog/2020-04-15-github-actions-new-workflow-features/

Workflow example

name: build
on: push
jobs:
  job1:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
    - id: set-matrix
      run: echo "::set-output name=matrix::{\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}"
  job2:
    needs: job1
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{fromJson(needs.job1.outputs.matrix)}}
    steps:
    - run: echo ${{ matrix.project }}
    - run: echo ${{ matrix.config }}

First job sets output variable matrix to JSON that contains two configurations:

{
  "include": [
    {
      "project": "foo",
      "config": "Debug"
    },
    {
      "project": "bar",
      "config": "Release"
    }
  ]
}

Equivalent in .yml:

  job2:
    strategy:
      matrix:
        include:
        - project: foo
          config: Debug
        - project: bar
          config: Release

Do not forget to escape quotes \" and print JSON in one line.

More complex Workflow example

It detects changed files and runs build job for changed directories. If directory name starts with OS name, it uses that name as runs-on.

name: Build
on: [push, pull_request]

jobs:

  generate-matrix:
    name: Generate matrix for build
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v2
      - name: Check changed files
        id: diff
        run: |
          # See https://github.community/t/check-pushed-file-changes-with-git-diff-tree-in-github-actions/17220/10
          if [ $GITHUB_BASE_REF ]; then
            # Pull Request
            git fetch origin $GITHUB_BASE_REF --depth=1
            export DIFF=$( git diff --name-only origin/$GITHUB_BASE_REF $GITHUB_SHA )
            echo "Diff between origin/$GITHUB_BASE_REF and $GITHUB_SHA"
          else
            # Push
            git fetch origin ${{ github.event.before }} --depth=1
            export DIFF=$( git diff --name-only ${{ github.event.before }} $GITHUB_SHA )
            echo "Diff between ${{ github.event.before }} and $GITHUB_SHA"
          fi
          echo "$DIFF"
          # Escape newlines (replace \n with %0A)
          echo "::set-output name=diff::$( echo "$DIFF" | sed ':a;N;$!ba;s/\n/%0A/g' )"
      - name: Set matrix for build
        id: set-matrix
        run: |
          # See https://stackoverflow.com/a/62953566/11948346
          DIFF="${{ steps.diff.outputs.diff }}"
          JSON="{\"include\":["

          # Loop by lines
          while read path; do
            # Set $directory to substring before /
            directory="$( echo $path | cut -d'/' -f1 -s )"

            if [ -z "$directory" ]; then
              continue # Exclude root directory
            elif [ "$directory" == docs ]; then
              continue # Exclude docs directory
            elif [ "$path" == *.rst ]; then
              continue # Exclude *.rst files
            fi

            # Set $os. "ubuntu-latest" by default. if directory starts with windows, then "windows-latest"
            os="ubuntu-latest"
            if [ "$directory" == windows* ]; then
              os="windows-latest"
            fi

            # Add build to the matrix only if it is not already included
            JSONline="{\"directory\": \"$directory\", \"os\": \"$os\"},"
            if [[ "$JSON" != *"$JSONline"* ]]; then
              JSON="$JSON$JSONline"
            fi
          done <<< "$DIFF"

          # Remove last "," and add closing brackets
          if [[ $JSON == *, ]]; then
            JSON="${JSON%?}"
          fi
          JSON="$JSON]}"
          echo $JSON

          # Set output
          echo "::set-output name=matrix::$( echo "$JSON" )"

  build:
    name: Build "${{ matrix.directory }}" on ${{ matrix.os }}
    needs: generate-matrix
    strategy:
      matrix: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v2
      - name: Build
        run: |
          cd ${{ matrix.directory }}
          echo "${{ matrix.directory }} ${{ matrix.os }}"

Answer 2

Adding a new example, it was a really helpful answer. Thank you @ArtemSBulgakov !

This one uses Github strategy.matrix of Github Actions with fromJson to collect only the directories in a Pull Request with changes and make a Syntax Review and Format Review of Terraform using https://github.com/dflook/terraform-github-actions

---
name: Check Syntax
on: [pull_request]

jobs:
  generate-matrix:
    name: Generate matrix for build
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Check changed files
        id: diff
        run: |
          # See https://github.community/t/check-pushed-file-changes-with-git-diff-tree-in-github-actions/17220/10
          export DIFF=$( git diff --dirstat=files,0,cumulative ${{ github.event.pull_request.base.sha }} | awk -F ' ' '{print $2}' )
          echo "$DIFF"
          # Escape newlines (replace \n with %0A)
          echo "::set-output name=diff::$( echo "$DIFF" | sed ':a;N;$!ba;s/\n/%0A/g' )"
      - name: Set matrix for build
        id: set-matrix
        run: |
          # See https://stackoverflow.com/a/62953566/11948346
          DIFF="${{ steps.diff.outputs.diff }}"
          JSON="{\"tfpaths\":["

          # Loop by lines
          while read path; do
          # Add item to the matrix only if it is not already included
          JSONline="\"$path\","
          if [[ "$JSON" != *"$JSONline"* ]]; then
          JSON="$JSON$JSONline"
          fi
          done <<< "$DIFF"

          # Remove last "," and add closing brackets
          if [[ $JSON == *, ]]; then
          JSON="${JSON%?}"
          fi
          JSON="$JSON]}"
          echo $JSON
          # Set output
          echo "::set-output name=matrix::$( echo "$JSON" )"

  validate:
    name: Check Terraform syntax on "${{ matrix.tfpaths }}"
    needs: generate-matrix
    strategy:
      matrix: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: terraform validate
        uses: dflook/terraform-validate@v1
        with:
          path: ${{ matrix.tfpaths }}

  check-format:
    name: Check Terraform format on "${{ matrix.tfpaths }}"
    needs: generate-matrix
    strategy:
      matrix: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: terraform fmt
        uses: dflook/terraform-fmt-check@v1
        with:
          path: ${{ matrix.tfpaths }}