Continue on error (but still report as error) in Azure Pipelines

Question

I have an Azure Pipelines pipeline defined by a YAML file that compiles, runs some tests, and then publishes test results. It's clearly impossible to run tests without compiling, so the compilation task obviously has the continueOnError: false set. However, I would still like to publish the test results when the tests fail, so I set continueOnError to true under the testing task.

This seemed like it worked, until one of my tests failed. Then, instead of failing the build, Azure just reported a warning. How can I get it to still error the entire build but also execute the remaining tasks?

Answer 1

UPDATE:

Microsoft has added some new documentation since I posted this answer that explains it better than I have:

A given task or job can't unilaterally decide whether the job/stage continues. What it can do is offer a status of succeeded or failed, and downstream tasks/jobs each have a condition computation that lets them decide whether to run or not. The default condition which is effectively "run if we're in a successful state".

Continue on error alters this in a subtle way. It effectively "tricks" all downstream steps/jobs into treating any result as "success" for the purposes of making that decision. Or to put it another way, it says "don't consider the failure of this task when you're making a decision about the condition of the containing structure".

---

ORIGINAL ANSWER:

I think it's useful to clear up a misunderstanding about exactly what a couple of the YAML properties do (because this got me too). The question asked about a YAML pipeline which the existing answer didn't cover so I thought I'd give an example of that too.

• continueOnError determines whether the current task continues if it encounters an error (with a warning), or if it fails straight away. Despite the name being potentially misleading, it does not (directly) determine whether execution continues to subsequent tasks or not.
• condition decides whether a task runs or not. By default, if a previous task failed, then this one will not run. You can override this and have tasks run regardless of earlier failures.

Therefore, it is not necessary to use continueOnError if your tests fail, just in order for the Publish Test Results task to run, you can have it run anyway.

I don't know exactly how your pipeline is structured, but hopefully this demonstrates an example:

trigger:
  - main

pool:
  vmImage: ubuntu-latest

steps:
- script: 'echo "compiling....."; exit 0'
  displayName: 'Compile'

- script: 'echo "testing....." ; exit 1'    # exit 1" simulates task failure 
  displayName: 'Run Tests'
  condition: succeeded()

- script: 'echo "publishing test results....."'
  displayName: 'Publish Results'
  condition: succeededOrFailed()            # Run task even if previous ones fail

The result is that the overall pipeline is failed:

And we can see the test results are still published in the pipeline breakdown (despite a previous step failing, there is no need here for continueOnError):

You can play around with the exit code in the YAML example to determine which tasks fail. I found the documentation of exactly what can go in the condition field to be a bit poor, but specifically I'd refer you to Conditions, Expressions, and Expressions#Job status check functions.

Answer 2

You can set the Publish test results task's Control options as below.

Then you will get the result as this.

If your compilation task always run successful, you can use the default conditions. If not, I think you can use custom conditions. For example, add a task to create a variable after your comilation task, then use the variable's value as the publish test results conditions, like and(succeeded(), eq(variables['variableName'], 'variableValue'))

You can specify the conditions under which the task or job will run. More detailed information, you can refer to here.