Python unit test that uses an external data file

Question

I have a Python project that I'm working on in Eclipse and I have the following file structure:

/Project
    /projectname
        module1.py
        module2.py 
        # etc.
    /test
        testModule1.py
        # etc.
        testdata.csv

In one of my tests I create an instance of one of my classes giving 'testdata.csv' as a parameter. This object does open('testdata.csv') and reads the contents.

If I run just this single test file with unittest everything works and the file is found and read properly. However if I try to run all my unit tests (i.e. run by right clicking the test directory rather than the individual test file), I get an error that file could not be found.

Is there any way to get around this (other than providing an absolute path, which I'd prefer not to do)?

Answer 1

Usually what I do is define

THIS_DIR = os.path.dirname(os.path.abspath(__file__))

at the top of each test module. Then it doesn't matter what working directory you're in - the file path is always the same relative to the where the test module sits.

Then I use something like this is in my test (or test setup):

my_data_path = os.path.join(THIS_DIR, os.pardir, 'data_folder/data.csv')

Or in your case, since the data source is in the test directory:

my_data_path = os.path.join(THIS_DIR, 'testdata.csv')

Edit: for modern python

from pathlib import Path

THIS_DIR = Path(__file__).parent

my_data_path = THIS_DIR.parent / 'data_folder/data.csv'

# or if it's in the same directory
my_data_path = THIS_DIR / 'testdata.csv'

Answer 2

Unit test that access the file system are generally not a good idea. This is because the test should be self contained, by making your test data external to the test it's no longer immediately obvious which test the csv file belongs to or even if it's still in use.

A preferable solution is to patch open and make it return a file-like object.

from unittest import TestCase
from unittest.mock import patch, mock_open

from textwrap import dedent

class OpenTest(TestCase):
    DATA = dedent("""
        a,b,c
        x,y,z
        """).strip()

    @patch("builtins.open", mock_open(read_data=DATA))
    def test_open(self):

        # Due to how the patching is done, any module accessing `open' for the 
        # duration of this test get access to a mock instead (not just the test 
        # module).
        with open("filename", "r") as f:
            result = f.read()

        open.assert_called_once_with("filename", "r")
        self.assertEqual(self.DATA, result)
        self.assertEqual("a,b,c\nx,y,z", result)