Python __init__.py vs sys.path.append/insert

Question

I know there are a ton of how-to import Python modules not in path, but I have yet to come across using Python's __init.py__ vs sys.path.insert. Which method is better? Are there any obvious drawbacks to either, like performance? Is one more "Pythonic?"

One scenario I can think of, is I have a program that users download and put in whatever directory, so I don't know the absolute path (unless I get it programatically). The folder structure is

working dir
    __init__.py
    foo.py
    src/
        my_utils.py
        __init__.py

I don't see any difference between using __init__.py or changing sys.path. Is there any scenario you can think of where it would make a difference?

Part 2 of my question is, why do you have to do anything to be able to import modules from subdirectories? I'm fairly new to Python, so maybe I'm not understanding why fiddling with the path or creating init files is boilerplate. To me, it seems like an unnecessary complication. If I have "dir" in my current working directory and say "import dir.my_utils," I don't see why I would have to list everything I would want to be able to import in __init__.py.

Apologies if this is a duplicate, but I did search before posting.

Edit: Here's another useful link: Automatically call common initialization code without creating __init__.py file

Answer 1

__init__.py is used by the python interpreter to treat directories as packages. Packages play an important role in avoiding namespace collisions. If you read the section 6.4 Packages from Python Modules, it helps to prevent directories with a common name from hiding other valid modules that occur later in the search path.

Hence, the package mechanism simplifies the task of importing packages. By using an __init__.py you can also do something like from package.subpackage import * which would be difficult or tedious to do if we were to keep appending to sys.path (in fact we will have to append all the possible modules).

As to answer the second part to your question - why do we need to do anything to treat directories as packages - well, there needs to be some way to tell python what should be allowed to import and what should not be. Also, you do not need to append anything to sys.path explicitly in case you have imported all the required modules at the beginning and that the modules you require to be imported are already present on the PYTHONPATH environment variable.

Hopefully this answer sheds some light on your query.