Running python unit-test cannot find my files in the correct directory to import

Question

This is what my code looks like in the code box below. When I run it the results for my_dir and network_json are, respectively:

my_dir: C:\Users\sepham\My Documents\LiClipse Workspace\sengtool_data_funnel

network_json:

C:\Users\sepham\My Documents\LiClipse Workspace\sengtool_data_funnel\example_EventsOfInterestColumnAggregate.json

But my project directory structure for the test_json_networks.py file is as follows:

C:\Users\sepham\Documents\LiClipse Workspace\sengtool_data_funnel\tests\test_networks

json_files_input = [
                'example_EventsOfInterestColumnAggregate.json',
                'example_EventsOfInterestConcatenate.json']
@pytest.mark.parametrize('network_json', json_files_input)
def test_network_jsons(network_json):

    my_dir = os.path.dirname(os.path.abspath(network_json))
    network_json = os.path.join(my_dir, network_json)

    print my_dir
    print network_json

    with open(network_json) as jdata:
        network = Network.load(json.load(jdata))

results of the print statement:

======================== CAPTURED OUTPUT =========================
C:\Users\sepham\My Documents\LiClipse Workspace\sengtool_data_funnel
C:\Users\sepham\My Documents\LiClipse 
Workspace\sengtool_data_funnel\example_EventsOfInterestColumnAggregate.json

The question is what environment variable or setting do I need to set up for Python unit-test so that it can see my file in correct directory at:

C:\Users\sepham\Documents\LiClipse Workspace\sengtool_data_funnel\tests\test_networks

By the way, i'm using the LiClipse IDE.

Also, this is the error message when running python unit-test because it cannot see my file to import:

E IOError: [Errno 2] No such file or directory: 'C:\\Users\\sepham\\My 
Documents\\LiClipse 
Workspace\\sengtool_data_funnel\\example_EventsOfInterestCo‌​
lumnAggregate.json' 

File "C:\Users\sepham\My Documents\LiClipse 
Workspace\sengtool_data_funnel\tests\test_networks\test_json‌​_networks.py", 
line 58 IOError

If i place the file to import in the directory as below, then it works correctly:

C:\Users\sepham\My Documents\LiClipse Workspace\sengtool_data_funnel

Answer 1

To summarise your issue:

You have the following project structure:

- sengtool_data_funnel (project root)
  - tests
    - test_networks
      - test_json‌​_networks.py
      - example_EventsOfInterestColumnAggregate.json
      - example_EventsOfInterestConcatenate.json

When you ran the test, the working directory was sengtool_data_funnel and not sengtool_data_funnel/tests/test_networks, where the JSON files are. Hence you encountered the exception.

Why is the project root the default working directory

Usually when you write a test, you will want to test your code that is within your project structure. Python will look in the working directory for modules (as well as the installed packages of course).

If your working directory is not the project root, then you might start to find it difficult to load any of your module not in that directory (e.g. in the parent directory). Even modules within the same directory can be problematic. That is because the absolute package structure no longer matches your project structure. And you may end up using hacks like amending the sys.path.

That is why the default working directory is the project root, and it should stay that way.

What can you do?

Now assuming the working directory is indeed the project root, we have several options.

First, you could simply load sengtool_data_funnel/tests/test_networks/example_EventsOfInterestConcatenate.json instead of example_EventsOfInterestConcatenate.json

Alternatively, you could load the file relative to your test script. Python defines __file__ for your test script. i.e. os.path.dirname(__file__) is would in that case point to sengtool_data_funnel/tests/test_networks. You could change your script to:

my_dir = os.path.dirname(__file__)
network_json = os.path.join(my_dir, network_json)

The advantage of that is that it would still work if you changed your working directory (although I wouldn't know why you would want to).