Correct setting of Python home and sys.prefix in an embedded environment

Question

Current situation

I embedded Python into my C++ application. The application is installed into the common Application locations on Windows, macOS (C:\Program Files, /Applications/, ...) which require admin privileges to make modifications to these directories. My application also ships with its own Python installation. Here is an example for the packaging structure:

C:\Program Files\Foo\foo.exe
C:\Program Files\Foo\Lib
C:\Program Files\Foo\DLLs
C:\Program Files\Foo\python.exe

The program also sets up an additional user-specific directory, which doesn't require any permissions and which is in sys.path:

C:\Users\username\AppData\Roaming\Foo\python\libs

Problem

During initialization I call PySet_Path, Py_SetPythonHome and I set sys.prefix to Program Files\Foo (as required by the documentation)

If the user installs pip or a module through pip it ends up in this application directory which would requires elevated permissions rights of course. But there is pip install something --user to find a user friendly installation directory (which is the AppData dir in my case).

My question is, which values and combinations should I set to make use of --user? Is it recommended to set the Python home to the AppData directory, and sys.prefix to the Program Files directory?

Because, that would mean the user could use --user to control where certain files are installed to

C:\Program Files\Foo\python.exe -m pip install xyz --user
would go into the user directory

C:\Program Files\Foo\python.exe -m pip install xyz
would go into the Program Files directory

Is this recommended, or am I overseeing any side-effects here?

Answer 1

I previously embedded a python interpreter into a Java application. I used Virtual Environments to manage things. This gives a lot of control over packages and where things end up.

Essentially you can have things in the system and/or override them in the virtualenv. And you can have virtualenv's for each user if needed.

See here for more about it:

https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/

Answer 2

As you have mentioned that the embedded system ships with its own python environment, thus it is not recommended to use pip to install libraries outside the environment.

That being said,

C:\Program Files\Foo\python.exe -m pip install xyz --user

The above command shall affect the system-wide behaviour of the Python environment for the user, which may lead to unexpected behaviour of the application as well as the Operating System. Also, this installation would mean that the installed application shall behave in a certain way for each user which may or may not be the same.

However,

C:\Program Files\Foo\python.exe -m pip install xyz

using this installation method helps in delivering similar behaviour of the application among users which is subjective as, if anything goes bad for one user, the application stops performing for all the users.