PHPUnit always output "No tests executed!" on MacOS Big Sur

Question

When trying to run any PHPUnit tests, I always get a No tests executed! message on my MacOS machine. A simple way to reproduce, on this specific machine, is to install a fresh instance of Laravel and running the default tests :

$ composer create-project --prefer-dist laravel/laravel blog
$ cd blog
$ vendor/bin/phpunit

=> No tests executed!

The expected output would be OK (2 tests, 2 assertions).

As far as I know, this is not a PHPUnit configuration issue since the default Laravel code is expected to work, the same issue appear with other framework and any code I try, the same issue is present with different PHPUnit version (8.5 and 9.4) and the exact steps listed above return the expected output inside an Ubuntu VM as well as on another Mac running Catalina.

Actually, I suspect it is not a PHPUnit issue, but more of a MacOS issue or a PHP configuration issue, one that could manifest in other form later with another tool or project.

PHPUnit used to work fine on this machine, but it's been a couples weeks/months I didn't actually used it. The only thing that changed since I last (successfully) used any PHPUnit on this Mac as been upgrading to MacOS Big Sur and installing (then uninstalling) Homebrew.

The issue appears PHPUnit can't find any testsuite. Running vendor/bin/phpunit --testsuite Unit still output No tests executed! while, inside a fresh Laravel install, should output Ok (1 test, 1 assertion).

So my question is : Is there anything else I can try to fix this before doing a fresh install of macOS Big Sur, and is anyone having the same issue?

EDIT - Dec. 1st 2020

As per requested in comments, here is my phpunit.xml which is, as described above, the default Laravel one.

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="./vendor/phpunit/phpunit/phpunit.xsd"
         bootstrap="vendor/autoload.php"
         colors="true"
>
    <testsuites>
        <testsuite name="Unit">
            <directory suffix="Test.php">./tests/Unit</directory>
        </testsuite>
        <testsuite name="Feature">
            <directory suffix="Test.php">./tests/Feature</directory>
        </testsuite>
    </testsuites>
    <coverage processUncoveredFiles="true">
        <include>
            <directory suffix=".php">./app</directory>
        </include>
    </coverage>
    <php>
        <server name="APP_ENV" value="testing"/>
        <server name="BCRYPT_ROUNDS" value="4"/>
        <server name="CACHE_DRIVER" value="array"/>
        <!-- <server name="DB_CONNECTION" value="sqlite"/> -->
        <!-- <server name="DB_DATABASE" value=":memory:"/> -->
        <server name="MAIL_MAILER" value="array"/>
        <server name="QUEUE_CONNECTION" value="sync"/>
        <server name="SESSION_DRIVER" value="array"/>
        <server name="TELESCOPE_ENABLED" value="false"/>
    </php>
</phpunit>

Again, this shouldn't matters, as any configuration returns the same result on this computer, even some that are known to work before, on other Mac and on other OS.

Running tests using artisan :

$ php artisan test --testsuite Unit
  No tests executed! 

  Time:   0.01s

Output of vendor/bin/phpunit --list-suites :

$ vendor/bin/phpunit --list-suites
PHPUnit 9.4.4 by Sebastian Bergmann and contributors.

Available test suite(s):
 - Unit
 - Feature

Output of vendor/bin/phpunit -v --testsuite Unit:

$ vendor/bin/phpunit -v --testsuite Unit
PHPUnit 9.4.4 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.3.22-(to be removed in future macOS)
Configuration: /Users/malou/Desktop/blog/phpunit.xml

No tests executed!

N.B.: /Users/malou/Desktop/blog/phpunit.xml is the one displayed above.

Referencing files directly works :

$ vendor/bin/phpunit ./tests/Unit
PHPUnit 9.4.4 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 00:00.006, Memory: 8.00 MB

OK (1 test, 1 assertion)

Edit #2 - Dec 1st 2020

More debug info for you :

$ php -v
WARNING: PHP is not recommended\nPHP is included in macOS for compatibility with legacy software.\nFuture versions of macOS will not include PHP.
PHP 7.3.22-(to be removed in future macOS) (cli) (built: Oct 30 2020 00:19:11) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.22, Copyright (c) 1998-2018 Zend Technologies
    with Xdebug v2.7.0, Copyright (c) 2002-2019, by Derick Rethans

MacOS Big Sur 11.0.1

Edit - Dec 2nd 2020

In response to PHPUnit always output "No tests executed!" on MacOS Big Sur:

I added the following:

    public static function main(bool $exit = true): int
    {
        var_dump(ini_get("auto_prepend_file"));
        var_dump($_SERVER['argv']); die;
        return (new static)->run($_SERVER['argv'], $exit);
    }

When executing vendor/bin/phpunit:

/Users/malou/Desktop/blog/vendor/phpunit/phpunit/src/TextUI/Command.php:163:
string(0) ""
/Users/malou/Desktop/blog/vendor/phpunit/phpunit/src/TextUI/Command.php:164:
array(1) {
  [0] =>
  string(18) "vendor/bin/phpunit"
}

When executing vendor/bin/phpunit --testsuite Unit

/Users/malou/Desktop/blog/vendor/phpunit/phpunit/src/TextUI/Command.php:163:
string(0) ""
/Users/malou/Desktop/blog/vendor/phpunit/phpunit/src/TextUI/Command.php:164:
array(3) {
  [0] =>
  string(18) "vendor/bin/phpunit"
  [1] =>
  string(11) "--testsuite"
  [2] =>
  string(4) "Unit"
}

Answer 1

tl;dr Apple broke version_compare by naming PHP 7.3.22-(to be removed in future macOS) in Big Sur. Installing another version of PHP fix this.

---

I found the answer to my issue. It is indeed a MacOS issue, related to the built in version of PHP in MacOS Big Sur.

After quite some debugging (on another project using PHPUnit 8), I endend up here : https://github.com/sebastianbergmann/phpunit/blob/ccbf3962a948112056b0eded6e4c880af4ee3695/src/Util/Configuration.php#L1041-L1055

    private function satisfiesPhpVersion(DOMElement $node): bool
    {
        $phpVersion         = \PHP_VERSION;
        $phpVersionOperator = '>=';

        if ($node->hasAttribute('phpVersion')) {
            $phpVersion = (string) $node->getAttribute('phpVersion');
        }

        if ($node->hasAttribute('phpVersionOperator')) {
            $phpVersionOperator = (string) $node->getAttribute('phpVersionOperator');
        }

        return \version_compare(\PHP_VERSION, $phpVersion, (new VersionComparisonOperator($phpVersionOperator))->asString());
    }

When looking at the last line of PHPUnit code above (and since both $node->hasAttribute('phpVersion') and $node->hasAttribute('phpVersionOperator') return false), the return statement can be simplified to:

version_compare(\PHP_VERSION, \PHP_VERSION, '>=')

Now, because the version of PHP that ships with MacOS got deprecated in Big Sur, Apple renamed the version to 7.3.22-(to be removed in future macOS). This is what caused the issue, as the above code now become :

version_compare("7.3.22-(to be removed in future macOS)", "7.3.22-(to be removed in future macOS)", '>=')

Which return false instead of true.

A simple way to test :

$foo = version_compare("7.3.22-(to be removed in future macOS)", "7.3.22-(to be removed in future macOS)", '>=');
var_dump($foo); // bool(false)

$bar = version_compare("7.3.22", "7.3.22", '>=');
var_dump($bar); // bool(true)

That is probably because, as explained in the official PHP Documentation...

The function first replaces _, - and + with a dot . in the version strings and also inserts dots . before and after any non number so that for example '4.3.2RC1' becomes '4.3.2.RC.1'. Then it compares the parts starting from left to right. If a part contains special version strings these are handled in the following order: any string not found in this list < dev < alpha = a < beta = b < RC = rc < # < pl = p. This way not only versions with different levels like '4.1' and '4.1.2' can be compared but also any PHP specific version containing development state.

Note that pre-release versions, such as 5.3.0-dev, are considered lower than their final release counterparts (like 5.3.0).

...the naming scheme of Apple is probably treated as a pre-release version inferior to itself instead of being equal to itself.

So the fix would to overwrite the php version, which can't be done globally as far as I know. Installing another version of PHP using Homebrew seems the easiest solution