Accessing resource files in Python unit tests & main code

Question

I have a Python project with the following directory structure:

project/
project/src/
project/src/somecode.py
project/src/mypackage/mymodule.py
project/src/resources/
project/src/resources/datafile1.txt

In mymodule.py, I have a class (lets call it "MyClass") which needs to load datafile1.txt. This sort of works when I do:

open ("../resources/datafile1.txt")

Assuming the code that creates the MyClass instance created is run from somecode.py.

The gotcha however is that I have unit tests for mymodule.py which are defined in that file, and if I leave the relative pathname as described above, the unittest code blows up as now the code is being run from project/src/mypackage instead of project/src and the relative filepath doesn't resolve correctly.

Any suggestions for a best practice type approach to resolve this problem? If I move my testcases into project/src that clutters the main source folder with testcases.

Answer 1

You can access files in a package using importlib.resources (mind Python version compatibility of the individual functions, there are backports available as importlib_resources), as described here. Thus, if you put your resources folder into your mypackage, like

project/src/mypackage/__init__.py
project/src/mypackage/mymodule.py
project/src/mypackage/resources/
project/src/mypackage/resources/datafile1.txt

you can access your resource file in code without having to rely on inferring file locations of your scripts:

import importlib.resources

file_path = importlib.resources.files('mypackage').joinpath('resources/datafile1.txt')
with open(file_path) as f:
    do_something_with(f)

Note, if you distribute your package, don't forget to include the resources/ folder when creating the package.

Answer 2

On top of the above answers, I'd like to add some Python 3 tricks to make your tests cleaner.

With the help of the pathlib library, you can explicit your ressources import in your tests. It even handles the separators difference between Unix (/) and Windows ().

Let's say we have a folder structure like this :

`-- tests
    |-- test_1.py <-- You are here !
    |-- test_2.py
    `-- images
        |-- fernando1.jpg <-- You want to import this image !
        `-- fernando2.jpg

You are in the test_1.py file, and you want to import fernando1.jpg. With the help to the pathlib library, you can read your test resource with an object oriented logic as follows :

from pathlib import Path

current_path = Path(os.path.dirname(os.path.realpath(__file__)))
image_path = current_path / "images" / "fernando1.jpg"

with image_path.open(mode='rb') as image :
    # do what you want with your image object

But there's actually convenience methods to make your code more explicit than mode='rb', as :

image_path.read_bytes() # Which reads bytes of an object

text_file_path.read_text() # Which returns you text file content as a string

And there you go !