How does one handle multiple modules/packages in python?

Question

I've spent hours researching this problem, and I'm still baffled. Please find my ignorance charming.

I'm building a python program that will allow me to pit two AIs against each other in a game of battleship.

Here's my directory structure:

.
├── ais_play_battleship
│   ├── game.py
│   ├── __init__.py
│   ├── player.py
│   └── ship.py
├── LICENSE
├── README.md
└── tests
    └── ship_test.py

2 directories, 7 files

Currently, I'm trying to write ship_test.py, but I cannot seem to import ais_play_battleship.ship. I get the dreaded "ModuleNotFoundError"

Here's what my research has taught me about my problem:

• If you want to import python code from another directory, you should make that directory a package instead of a module. Therefore, I've placed an __init__.py file in the root of ais_play_battleship.
• Python will only search the directory python is launched from as well as the directory of the script you're running. Therefore, I've been trying to launch my tests by running python3 tests/ship_tests.py from the root directory.

Here are my specific questions:

• Why is the error a "ModuleNotFound" error? Shouldn't it be "PackageNotFound"?
• Am I correct to make ais_play_battleship a package?
• How can I keep my tests in a separate directory and still use the code in ais_play_battleship?

Please forgive me, as I'm not very good at asking questions on StackOverflow. Please tell me how I can improve.

Answer 1

I am answering my own question, as I haven't yet received a satisfactory answer. The best resource I've found is available here. In summary:

Python does NOT search the directory you run python from for modules. Furthermore, adding an __init__.py file to make a directory a package is not enough to make it visible to an instance of python running in another folder. You must also install that package. Therefore, the only two ways to access a module in another directory are:

1. Install the packaged module in site-packages (this requires the creation of a setup.py file and installation using pip install . More information is available here.
2. Modify path to resolve the module

I ended up settling with the second option, for reasons discussed below.

The first option requires one to reinstall the package at every change to a package. This is difficult on a constantly-changing codebase, but can be made easier by using build automation. However, I'd like to avoid this added complexity.

I shied away from the second option for a long time, because it seemed that modifying the path would require hard-coding the absolute path to my module, which is obviously unacceptable, as every developer would have to edit that path to fit their environment. However, this guide provides a solution to this problem. Create a ./tests/context.py file with the following contents:

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

Then, in my ship_test.py module, I imported the context and the module I needed:

import context
import ais_play_battleship.ship
# (I include the submodule ship because ais_play_battleship itself does not have
# any attributes or methods, and the submodule ship is the only one I am testing
# in ship_test.py)

This fits my project better, because it works as expected without having to worry about installing my package (or the method by which my package was installed).

Answer 2

To solve this problem without relying on hacking about your sys.path, create a setup.py file and as a build step for your test runner, have it run pip install . first. You might want to use a tool like tox.

In the top level directory:

setup.py

from setuptools import setup

setup(name='ais_play_battleship')

tox.ini

[tox]
envlist = py36, py37

[testenv]
deps=pytest
commands=
    pip install . --quiet
py.test -q

then run your tests (in this example we use tox to do this so that we can also configure how the test environment can be configured) : tox