phpunit & paratest & Laravel - Random failures in creating the test storage directory

Question

I am using parallel testing addon for phpunit, paratest, with a Laravel application to speed up the execution of our testsuite. This works most of the time but occasionally I get the following failure.

League\Flysystem\Exception: Impossible to create the root directory "/codebuild/output/src0123456/src/github.com/org/repo/storage/framework/testing/disks/local". file_get_contents(/codebuild/output/src0123456/src/github.com/org/repo/.env): failed to open stream: No such file or directory

/codebuild/output/src0123456/src/github.com/org/repo/vendor/league/flysystem/src/Adapter/Local.php:112
/codebuild/output/src0123456/src/github.com/org/repo/vendor/league/flysystem/src/Adapter/Local.php:78
/codebuild/output/src0123456/src/github.com/org/repo/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemManager.php:167
/codebuild/output/src0123456/src/github.com/org/repo/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:261
/codebuild/output/src0123456/src/github.com/org/repo/vendor/laravel/framework/src/Illuminate/Support/Facades/Storage.php:70
/codebuild/output/src0123456/src/github.com/org/repo/tests/TestCase.php:42

The failure on line 42 relates to this line which is creating the local storage folder for testing.

Storage::persistentFake();

I think the second half of the error that mentions the .env file is unrelated as the exception picks the last logged error rather than the error related to the failure.

This only happens every now and again so it must be a sequence of operation or timing issue.

The tests are running and failing inside an AWS codebuild environment against php 7.3 and 7.4.

Anyone have any ideas?

Answer 1

Incase anyone else comes across this, it was resolved by creating the test storage directory before executing the tests.

mkdir -p storage/framework/testing/disks/local
vendor/bin/paratest

It's a little brittle but so far has worked perfectly for us.

Answer 2

From my experience this is usually not an issue with the file system. Most of the times I had a test not cleaning up correctly.

Depending on the file system and Paratest your tests are executed in a different order and then this errors happens.

There are a few things you can do to track this down:

1. Enable --debug mode when executing the tests on your build environment and check all the test that were executed before.
2. As of PHPUnit 7.3 use --order-by=random locally to execute your tests in a different order then they appear when reading from the file system. Execute it a few times, maybe you can then simulate this locally.
3. I see that you're using Flysystem: try to execute the tests with the Memory filesystem adapter to make sure it is really not a file system problem.
4. Make sure that every test creates the filesystem layout before (Testcase::setUp()) and cleans it when shut down (TestCase::teatDown()), otherwise one test can have an influence on another.
5. Make sure your tests don't depend on values that may change. For example I had problems with tests that involved dates and I've executed tests on Jenkins at 23:59 and they failed, because the date switched to the next day. In those cases pass a date to work with through the test.