Differences between github API commit statuses "failure" and "error"?

Question

What is the difference between github API commit statuses "failure" and "error"?

Answer 1

The GitHub API Statuses do include marking commits with a success, failure, error, or pending state.

Typically, in a continuous integration context, a commit is:

• marked as failed because the job failed to complete
• marked as error because the job did complete, but exited with a non-zero status
• marked as success because the job did complete, and exited with a zero status

(job as tasks run by job scheduler)

---

Since 2014, the integration of GitHub with CI did evolve.
In May 2018, you had "Introducing the Checks API, a better way to connect integrations and code".
It introduced the notion of checks

Rather than binary pass/fail build statuses, integrations can report rich statuses, annotate lines of code with detailed information, and kick off reruns.

You find failure in that new context (of checks)

When someone pushes code to a repository, GitHub creates a check suite for the last commit. GitHub Apps with the checks:write permission receive a check_suite webhook with the requested action. When your GitHub App receives the check_suite event, it can create new check runs for the latest commit.

This works with:

• Travis-CI (which has since then deprecated the GitHub commit status API updates for GitHub Apps-managed repositories)
• Circle-CI

In that new context (New Checks API public beta):

• failure is associated with a check which runs, but exit with a non-zero status,
• error is associate with a check which cannot even run

See Questions:

How are check runs different than commit statuses?

Commit statuses allow for a simple pass or fail state.
Check runs allow for much more granular information: they can conclude as either success, failure, neutral, cancelled, timed_out, or action_required. Check runs are more flexible than commit statuses.

---

From the comments

Juraj Martinka:

In unit testing terminology:

• "failure" usually represents a failed test, that is the code under the test returned an answer but this is different from the expected one;
• "error" represents an unexpected error in the code like an exception being thrown.

This is also what I've seen to be used in CI such as Travis:

• "build failed" means that your change breaks some tests, has linter errors, etc.
• "build error" means unexpected build job error (early termination).

Bob Bell agrees:

I use:

• "failure" to say "the test failed against the criteria", and
• "error" to be "the test was not able to be executed properly".

I consider "failure" and "success" to be opposite, the two results of a completed test, while "error" is the outlier, the result of an aborted test.

Answer 2

I asked this question from official GitHub and received next response:

Hi Alexander,

Thanks for contacting GitHub Support!t

• A failed CI check is when the check has not passed the required conditions.
• A CI check that is in error is when the check itself has an error that prevents it from running correctly.

Let me know if you need any further information.