What form of imports should I use in __main__.py and then how should I run the project?

Question

Suppose I have the following simple project structure.

project/
    package/
        __init__.py
        __main__.py
        module.py
    requirements.txt
    README.md

After doing some research on Google, I've tried to make it reflect general best practices for a very simple console application (but not as simple as simply having a single script). Suppose __init__.py contains simply print("Hello from __init__.py") and module.py contains a similar statement.

How should I do imports inside of __main__.py, and then how should I run my project?

Let's say first that __main__.py looks simply like this:

import module
print("Hello from __main__.py")

If I run my project with the simple command python package, I get this output:

Hello from module.py
Hello from __main__.py

As can be seen, __init__.py didn't run. I think this is because I'm running my project's package as a script. If I instead run it as a module with the command python -m package I get this output:

Hello from __init__.py
Traceback (most recent call last):
  File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\MY_USER\PATH\TO\PROJECT\project\package\__main__.py", line 1, in <module>
    import module
ModuleNotFoundError: No module named 'module'

If I change the first line in __main__.py to import package.module and run python -m package again I get this output:

Hello from __init__.py
Hello from module.py
Hello from __main__.py

Great! It seems everything runs properly now when running my project's package as a module. Now what if I try python package again and run it as a script?

Traceback (most recent call last):
  File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "package\__main__.py", line 1, in <module>
    import package.module
ModuleNotFoundError: No module named 'package'

Alright. So please correct me if I'm wrong, but it seems that I have two options. I can either write the imports in my package to work with running it as a script or running as a module, but not both. Which is better, if one is indeed preferable, and why? When would you use the command python package vs python -m package, and why? Is there some general rule for writing imports within a simple project that I might not be understanding? Am I missing something else fundamental?

In summary: What is the best practice in this situation, why is it the best practice, and when would you set your project up for the alternative approach (python package vs python -m package)?

Answer 1

The most common way to distribute executable python code is to package it into an installable .wheel file. If it's only a single file, you can also just distribute that. But as soon as you got two files, you run into the exact import issues that you experience, at which point you need some metadata to have a well-defined and visible toplevel for imports (which would e.g. be package.module in your case), entrypoints for script code, third party dependencies... all of which is achieved by "making the code installable".

If you like technical documentation, you can read up on what exactly that means by going through this and this tutorial by the python packaging authority (PyPA).

---

To get you started with your project though, what you are missing is a setup.py file which will contain install instructions and a script entrypoint to provide a way to run executable code from within your package:

from setuptools import setup

with open("requirements.txt") as f:
    requirements = [line.strip() for line in f.readlines()]


setup(
    # obligatory, just the name of the package. you called it "package" in your
    # example so that's the name here
    name="package",
    # obligatory, when it's done you can give it a 1.0
    version="0.1",
    # point the installer to the module (read, folder) that contains your code,
    # by convention usually the same as the package name
    packages=["package"],
    # if there are dependencies, specify them here. actually you can delete the
    # requirements.txt and just paste the content here, but this here will also work
    install_requires=requirements,
    # point the installer to the function that will run your executable code.
    # the key name has got to be 'console_script', the value content is up
    # to you, with its interpretation being:
    # 'package_command' -> the name that you can call your code by
    # 'package.__main__' -> the path to the file that you want to call
    # 'test' -> the actual function that contains the code 
    entry_points={'console_scripts': ['package_command=package.__main__:test']}
)

After adding this file as project/setup.py, you need to keep the following things in mind:

• put your script code (e.g. print('hello world)') into a function called test within your __main__.py file^[1]
• run pip install -e . to install your package locally^[2]
• execute the script code by running package_command on the command line - neither python package nor python -m package is best practice to run installable python scripts

---

^{[1] _{Binding a simple function to a script-entrypoint is as basic as it gets. You probably don't want to re-invent things like help-texts, argument parsing/verifying, ... so if you really mean to write a cli-application, you might want to look into something like click to handle the tedious stuff for you.}}

^{[2] _{This performs a dev-install, which is convenient during development since it means that you can test behavior as you work on it. If you want to distribute your code, you'd run pip wheel . (might require running pip install wheel before). This will create a wheel file that you can give to your clients so they can install it manually with pip install package-0.1-py3-none-any.whl, after which package_command would also be available on their systems.}}