Python relative import: ModuleNotFoundError: No module named 'sib1'

Question

I have a project hierarchy like below, when I run python src/bot/main I didn't get error. While if I run python -m src.bot.main I got an error. Why?

This is my file hierarchy:

MyProject
└── src
    ├── __init__.py
    ├── bot
    │   ├── __init__.py
    │   ├── main.py
    │   └── sib1.py
    └── mod
        ├── __init__.py
        └── module1.py

This is the content of main.py:

import sys
    

if __name__ == "__main__":

    # sys.path will print the parent folder.
    print(sys.path, end="\n\n")
    
    # my problem is on this line.
    import sib1
    sib1.test()
    

The error:

Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/me/Desktop/test_py/src/bot/main.py", line 16, in <module>
    import sib1
ModuleNotFoundError: No module named 'sib1'

Some conclusion I've made so far:

Since the output of sys.path in both cases include /Users/me/Desktop/MyProject, the reason should not related to scope?

---

The output of sys.path of both python -m src.bot.main and python src/bot/main:

(test) ✔ me Desktop/test_py % python -m src.bot.main
['/Users/me/Desktop/test_py', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python39.zip', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/lib-dynload', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/site-packages']

Answer 1

I will try my best to clarify each of my confusions in a Q&A form, and organize @Brain's comments along the way:

---

Q1: when I run python src/bot/main I got no error.

The sys.path will include the current directory containing the file main.py, i.e. the interpreter will see the file MyProject/src/bot:

import sib1

is logically equivalent to:

import "MyProject/src/bot" + "/sib1.py"

Hence, no error.

---

Q2: While if I run python -m src.bot.main I got error. Why?

Now it's time to quote @Brain's valuable (first) comment:

Using python -m src.bot.main tells Python that src is a top-level package. Everything below src in the directory structure will be considered submodules/subpackages of src. The proper name for sib1 under that organization is src.bot.sib1. No top-level module named sib1 exists as far as Python is concerned.

(emphasis by me)

So:

• The -m option will define the scope (and thus the top-level package) for the file you're going to run.
• All packages not starting from src. will be regarded as built-in or third-party libraries installed in your virtual environment. Since I didn't install any package called sib1, you got the error. (the sys.path is intentionally ignored in this case, as we there is no implicitly relative import starting from Python 3.3)
• All your relative imports should not go outside of the top-level package src (by prepending too many .'s in the import).

For example, this will work:

from . import sib1
from ..mod import module1    # The `..` is equivalent to the MyProject/src.

module1.hello()
sib1.test()

---

Finally, don't test your package by inserting many if __name__ == '__main__'. Do so by professional tools:

If you only need to run submodules for testing purpose, you could consider using more robust testing tools like doctest (lightweight) or unittest (heavyweight).

This is a good read I had put 300 bounty on it years ago, you must find a time to read it.