Python can import a module that isn't installed

Question

So, I'm playing around with packaging a python script I've written, and it has a submodule, let's call it submodule. The folder structure looks like this:

cool_script/
  setup.py
  cool_script.py
  submodule/
    __init__.py
    implementation.py

Now, after many pip install . and pip install -e . calls, I have the situation where submodule can be imported globally. No matter where on my system, this will always work:

$ python3
[...]
>>> import submodule
>>> submodule.__file__
'/home/me/fake/path/cool_script/submodule/__init__.py'

But I don't know why.

The package I made was uninstalled again, and pip cannot find the submodule in its index. There's nothing in dist-packages either, I manually deleted the cool_script.egg-link that was still sitting around there:

$ ls /usr/local/lib/python3.4/dist-packages | ack cool 
$ ls /usr/local/lib/python3.4/dist-packages | ack submodule
$ 

The PYTHONPATH is empty as well:

$ echo $PYTHONPATH

$

Why does Python know the location of submodule? How can I find out?

Answer 1

First run python -c "import site; print(site.getsitepackages())". It will print a list like this:

['/XXX/something/site-packages']

Normally there is a single path in this list, and it points to a directory where pip installs your scripts. You can ls into it if you're curious: ls /XXX/something/site-packages/.

More interestingly, though, pip puts a "link" file in that directory when you're using developer installs (a.k.a. pip install -e). The "link" file is named after the original project with a .egg-link extension at the end.

So you probably have a cool_script.egg-link file in that directory. And if you try to print it out you should find that its contents list the original filesystem location of your module. Something like:

$ cat /XXX/something/site-packages/cool_script.egg-link
/home/me/fake/path/cool_script/
.

This is how pip records that it has installed something in developer mode, but it isn't how Python actually knows how to find your module (that would have been too easy, right? :-)).

Python doesn't know about .egg-link files, but it reads all .pth files in the site-packages directory to get additional paths for sys.path (*). So, for Python to be able to import developer mode installs, pip writes all theirs paths in a single .pth file conventionally called easy-install.pth (because the old easy-install tool actually pioneered that technique). And if you print out that file, you'll get the list of all projects paths installed in developer mode:

$ cat /XXX/something/site-packages/easy-install.pth
/home/me/fake/path/cool_script/
/home/me/another/project/

And you can check that indeed all those paths listed in easy-install.pth indeed get added to your sys.path.

(*) Technically, the part of Python that reads those .pth file is the site module which is normally imported automatically at startup. There is an option to disable the site module, though, for example by using python -S. In that case, you'll see that sys.path contains neither the site-packages directory nor the developer install paths.