Why are .pth files loaded twice in a Python virtual environment?

Question

In the user site .pth files are processed once at interpreter startup:

$ echo 'import sys; sys.stdout.write("hello world\n")' > ~/.local/lib/python3.8/site-packages/hello.pth
$ python3.8 -c ""
hello world

And it's the same behaviour in the system site, e.g. /usr/local/lib/python3.8/site-packages/.

But in venv they are processed twice:

$ rm ~/.local/lib/python3.8/site-packages/hello.pth
$ /usr/local/bin/python3.8 -m venv .venv
$ source .venv/bin/activate
(.venv) $ echo 'import sys; sys.stdout.write("hello world\n")' > .venv/lib/python3.8/site-packages/hello.pth
(.venv) $ python -c ""
hello world
hello world

Why are path configuration files processed twice in a virtual environment?

Answer 1

Looks like it all happens in the site module (not so surprising). In particular in the site.main() function.

The loading of the .pth files happens either in site.addsitepackages() or in site.addusersitepackages(), depending in which folder the file is placed. Well more precisely both these function call site.addpackage(), where it actually happens.

In your first example, outside a virtual environment, the file is placed in the directory for user site packages. So the console output happens when site.main() calls site.addusersitepackages().

In the second example, within a virtual environment, the file is placed in the virtual environment's own site packages directory. So the console output happens when site.main() calls site.addsitepackages() directly and also via site.venv() a couple of lines earlier that also calls site.addsitepackages() if it detects that the interpreter is running inside a virtual environment, i.e. if it finds a pyvenv.cfg file.

So in short: inside a virtual environment site.addsitepackages() runs twice.

As to what the intention is for this behavior, there is a note:

        # Doing this here ensures venv takes precedence over user-site
        addsitepackages(known_paths, [sys.prefix])

Which, from what I can tell, matters in case the virtual environment has been configured to allow system site packages.

Maybe it could have been solved differently so that the path configuration files are not loaded twice.