Travis CI : How to allow failures with a customized environment variable?

Question

Following this answer, I wrote this Travis configuration file :

language: php  php:     - 5.3     - 5.4     - 5.5     - 5.6     - 7     - hhvm     - nightly  branches:     only:         - master         - /^\d+\.\d+\.\d+$/  matrix:     fast_finish: true     include:         - php: 5.3           env: deps="low"         - php: 5.5           env: SYMFONY_VERSION=2.3.*         - php: 5.5           env: SYMFONY_VERSION=2.4.*         - php: 5.5           env: SYMFONY_VERSION=2.5.*         - php: 5.5           env: SYMFONY_VERSION=2.6.*         - php: 5.5           env: SYMFONY_VERSION=2.7.*         - php: 5.5           env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail     allow_failures:         - php: nightly         - env: TEST_GROUP=canFail  before_script:     - composer self-update     - if [ "$SYMFONY_VERSION" != "" ]; then composer require --dev --no-update symfony/symfony=$SYMFONY_VERSION; fi     - if [ "$deps" = "low" ]; then composer update --prefer-lowest; fi     - if [ "$deps" != "low" ]; then composer update --prefer-source; fi  script: phpunit 

But Travis CI counts only the php nightly version as an "allowed to fail" version. Am I using the environment variables in a wrong way ?

---

UPDATE

Just a precision, I know that I can directly write the environment like that:

matrix:     include:         - php: 5.5           env: SYMFONY_VERSION=2.8.*@dev     allow_failures:         - env: SYMFONY_VERSION=2.8.*@dev 

but still I don't get why the other way doesn't work.

Answer 1

What you specify in allow_failures: is your allowed failures

"You can define rows that are allowed to fail in the build matrix. Allowed failures are items in your build matrix that are allowed to fail without causing the entire build to fail. This lets you add in experimental and preparatory builds to test against versions or configurations that you are not ready to officially support."

Unfortunately, I believe the way the matrix reads your first set of code is as php nightly version as the "allowed to fail" version with the environment as part of nightly.

Because of how Travis allows failures it must be an exact match, you can not just specify env: as an allowed failure you have to specify for each php version with the env: you want to allow as a failure for example

allow_failures:   - php: 5.3     env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail   - php: 5.4     env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail   - php: 5.5     env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail   - php: 5.6     env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail   - php: 7.0     env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail   - php: hhvm     env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail   - php: nightly # Allow all tests to fail for nightly 

Answer 2

According to this issue, the php and env keys must match perfectly. env can be either a single value or an array, but in both case it must be a perfect match. So if you want your build:

- php: 5.5   env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail 

to be allowed to fail, you have to either give the whole env key SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail and the whole env key and the PHP version (if you had the same env key for different PHP versions).