Merging coverage results from multiple Azure Pipeline jobs in Python (with pytest)

Question

I've setup my open source project to run CI with Azure Pipelines, and am collecting code coverage following the example from the Azure pipelines docs on how to test Python apps.

This seems to work pretty well, but the code coverage statistics seem to only pick up a test results from a single job (at random). To get complete coverage for my project (e.g., for platform dependent code), I really need to aggregate coverage across all of the test jobs.

Here are the relevant tasks from my pipeline:

- bash: |
    source activate test_env
    pytest xarray --junitxml=junit/test-results.xml \
    --cov=xarray --cov-config=ci/.coveragerc --cov-report=xml
  displayName: Run tests
- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'
    reportDirectory: '$(System.DefaultWorkingDirectory)/**/htmlcov'

What's the right way to configure Azure to show this information?

I've tried adding --cov-append into my pytest invocation but that doesn't seem to make a difference.

Answer 1

As per documentation, when you have multiple tests running from different jobs, the results are cleaned after each job, so if you want to gather all the tests results into one place you must publish the results to artifacts and at the end of all test jobs, you'll have to consolidate them.

Basically for each test you'd have:

- jobs:
  - job: testjob1
    steps:
    - step1 -> run tests
    - step2 -> publish job1 artifacts
  - job: testjob2
    steps:
    - step1 -> run tests
    - step2 -> publish job2 artifacts
   ....
  - job: consolidate
    steps:
    - step1 -> downloadArtifacts
    - step2 -> publish test results

at the end You'll need a job to download all these artifacts, them publish them exactly as you did.

ref.: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml

ref.: https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts?view=azure-devops&tabs=yaml