Module can't be found when called from outside

Question

I have a multiple sub-modules python project. For example, "myproject" which has two submodules namely "submodule1" and "submodule2".

Project structure

Example1/
|-- submodule1/
|   |-- __init__.py
|   |-- hello.py
...
|-- submodule2/
|   |-- __init__.py
...

And I have a hello.py in submodule1 which has the following content:

import datetime

def greeting():
    print("hello world! - " + datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))

The content for submodule1's __init__.py is:

def main():
    import hello
    hello.greeting()

if __name__ == '__main__':
    main()

I am able to import hello.py and call greeting() function successfully here.

Now, I installed submodule1 as a package to python, and then created submodule2, and tried to call submodule1 from submodule2 by putting following code in __init__.py from submodule2:

import submodule1

def main():
    submodule1.main()

if __name__ == '__main__':
    main()

It fails with this message:

....in main submodule1.main()
    ModuleNotFoundError: No module named 'hello'

I think hello shouldn't be exposed to outside, how am I able to let my submodule1 works? I'm using python3

========

setup.py is:

from setuptools import setup, find_packages

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

__version__ = "0.0.2"

setup(
    version=__version__
    name = "mymodules",
    packages = find_packages(exclude=['tests', '*.tests', '*.tests.*']),
    install_requires=requirements,
)

Answer 1

Use

def main():
    from .submodule1 import hello 
    # or from Example1.submodule1 import hello
        hello.greeting()

if __name__ == '__main__':
    main()

It is not finding hello because it is not in submodule2 folder. You have to give the interpreter the relative path inside the package, or the absolute path in the package. This only works with the from ... import syntax form.

Omitting the package name can be useful if you have lots of imports and later want to change your package name.

Edit:

This is from the point of view of the code inside the Example1 package, if you turn it into one. That is, this is the way code inside the package Example1, in some submodule, can refer code inside other Example1 submodule, with submodules placed in different folders also with their __init__.py inside. This is insider knowledge, but it is to be used from inside the package Example1. When you are outside, importing the package Example1, you can import all the names you want to be seen from the outside (in the importer) in the Example1 __init__.py file.

So I see two different questions here:

How do I refer code from submodule to submodule, with both being inside the same package (do like I said), and how do I export names inside my package to outsiders importing my package (import the names in the package _init__.py so you can write things like from MyPackage import Class1, Class2, other_name or even * - use an __all__ list to control those names).

Also I don't understand why you call submodules to your submodules and then try to install one of them as an isolated package and use it like that. Aren't they part of your Example1 package? If they were isolated packages they shouldn't refer the others...

Stated like this, this may look confusing, actually packages aren't the simplest concept to explain, but it is not that difficult either.

See a clean example I made here on how to organize classes as a package and use them inside the package (insider knowledge of course). Pay attention to the __init__.py files content as it is similar to what you want to do to make names available outside your package (no inside knowledge except for the importable names).

2nd Edit:

So I trace some source of confusion between me and OP to this: everything I said above applies to import packages while OP is thinking in terms of distribution packages. Not surprisingly, even the glossary refers this confusion as a real possibility.

Did you try

from . import hello

in the submodule1 import package __init__.py, instead of import hello? That should use the submodule1 namespace.

3rd edit:

Your errors might happen because after setting up imports, running with sucess still depends on how you are running the code (I mean from where you import the whole thing), which is usually the package parent folder. If you try to start code from an inside point it can still fail despite being correct (sorry the incoherence).

Even that can be solved with absolute imports but that requires Example1 to become a package, because you have dependencies between the submodules (actually subpackages).

And that leads me to another point:

An import package can contain nested import packages (subpackages), and also modules. That makes it adequate for any project. That means also there is an outer package that contains everything. That is your project. Make releases of that outer package, your project.

Suppose you say "submodules have to be installed separately". Fine. Assuming they are independent, make them packages (they are packages already). Release them separately as above.

Or,

to me they look related, it looks some "submodules" depend on others.

You also say "submodules underneath can be responded by different developers". That is no excuse. It looks to me as a common code base, put them all in version control (you did, didn't you). Tag the files in release points, or put releases in their own branch. Make it all a single package. Release that package.

Otherwise I think you are setting up for chaos. And you are trying to use release packages to control it. How do you keep a log of what version works with what version of another submodule. And if you put the sources of different submodules together for test, they will conflict (override) previously installed submodules in your system.

Also your package will have a single entry point, either a simple import of it or some main hook to run a main function perhaps, not several possible entry points from each "submodule".

I hope that this helps solving your problem, not exactly your error.

Answer 2

For Python3.5 and assuming you want to install packages submodule1 and submodule2, if not please mention how you intend to use them: This is how the project is structured

    (2005env) python@python:~/2005env/Example1$ tree
.
├── requirements.txt
├── setup.py
├── submodule1
│   ├── hello.py
│   └── __init__.py
└── submodule2
    └── __init__.py

2 directories, 5 files

Placed setup.py outside both submodules, it can also be placed inside individual submodules to install them separately with a bit of change in directory structure and installation.

(2005env) python@python:~/2005env/Example1$ pip install -e .
Obtaining file:///home/python/2005env/Example1
Installing collected packages: submodule1
  Running setup.py develop for submodule1
Successfully installed submodule1

Now importing submodule2 in python3.5 from location outside project directory i.e. /home/python and project is in /home/python/2005env/Example1

(2005env) python@python:~$ python
Python 3.5.3 (default, Nov 23 2017, 11:34:05) 
[GCC 6.3.0 20170406] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import submodule2
>>> dir(submodule2)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'main', 'submodule1']
>>> submodule2.main()
Hello World! - 2018-02-25 09:03:32
>>> dir(submodule2.submodule1)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'greeting', 'hello', 'main']


Python 3.5.3 (default, Nov 23 2017, 11:34:05) 
[GCC 6.3.0 20170406] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from submodule1 import greeting
>>> greeting()
Hello World! - 2018-02-25 09:26:14

In init.py of submodule1 I have imported hello as well as greetings hence those are available as attributes

(2005env) python@python:~/2005env/Example1$ cat submodule1/__init__.py 
from . import hello
from .hello import greeting


def main():
    hello.greeting()

if __name__ == '__main__':
    main()

setup.py is same as yours with find_packages but name if submodule1 instead of mymodules

Answer 3

Finally, I figured it out. I can't put

if __name__ == '__main__':
    main()

in the __init__, it never work, which caused me try to run it directly and failed by ImportError: cannot import name 'hello'. When __init__ is __main__, it cannot import substuffs, but it works if called from outside, for example, from submodule2, submodule1.main() still available. So this issue looks doesn't impact to user now. Thanks for @progmatico, you answers help me a lot to understand how those functions work together, very appreciate!