Failed to import python module from different directory

Question

I have this code structure in python3:

- datalake
  __init__.py
  utils
     __init__.py
     utils.py
  lambdas
     __init__.py
     my-lambdas.py
- tests
    __init__.py
    demo.py

All init__.py files are empty.

My problem is how I can import datalake module from tests/demo.py? I tried from datalake.utils import utils in demo.py but when I run python tests/demo.py from command line, I get this error ModuleNotFoundError: No module named 'datalake'.

If I use this code:

from ..datalake.utils import utils

I will get error ValueError: attempted relative import beyond top-level package.

I also tried to import the module utils from my-lambda.py file which also failed. The code in my-lambda.py is from datalake.utils import utils but I get ModuleNotFoundError: No module named 'datalake' error when run python datalake/lambda/my-lambda.py from command line.

How can I import the module?

Answer 1

When you run a command like python tests/demo.py, the folder you are in does not get added to the PYTHONPATH, the script folder does. So a top-level import like import datalake will fail. To get around this you can run your tests as a module:

Python 2:

python -m tests/demo

Python 3:

python -m tests.demo

and any datalake imports in demo.py will work.

It sounds like what you really want to do is have a folder with tests separate to your main application and run them. For this I recommend py.test, for your case you can read Tests Outside Application Code for how to do it. TL;DR is run your tests from your top level project folder with python -m py.test and it will work.

Answer 2

First of all, my-lambdas.py is not importable with the import statement as hyphens are not valid in Python identifiers. Try to follow PEP-8's naming conventions, such as mylambdas.py.

Otherwise the package structure looks good, and it should be importable as long as you are at the level above datalake/, e.g., if you were in the directory myproject/ below:

myproject
├── datalake
│   ├── __init__.py
│   ├── utils
│   │   ├── __init__.py
│   │   └── utils.py
│   └── lambdas
│       ├── __init__.py
│       └── mylambdas.py
└── tests
    ├── __init__.py
    └── demo.py

Then this should work:

~/myproject$ python -c 'from datalake import utils'

Otherwise, setting the environment variable PYTHONPATH to the path above datalake/ or modifying sys.path are both ways of changing where Python can import from. See the official tutorial on modules for more information.

Also some general advice: I've found it useful to stick with simple modules rather than packages (directories) until there is a need to expand. Then you can change foo.py into a foo/ directory with an __init__.py file and import foo will work as before, although you may need to add some imports to the __init__.py to maintain API compatibility. This would leave you with a simpler structure:

myproject
├── datalake
│   ├── __init__.py
│   ├── utils.py
│   └── lambdas.py
└── tests
    ├── __init__.py
    └── demo.py