Python Shared Libraries

Question

As I understand there are two types of modules in python (CPython): - the .so (C extension) - the .py

The .so are only loaded once even when there are different processes/interpreters importing them.

The .py are loaded once for each process/interpreter (unless reloading explicitly).

Is there a way .py can be shared by multiple processes/interpreters?

One would still need some layer where one could store modifications done to the module. I'm thinking one could embed the interpreter in a .so as the first step. Is there an already developed solution.

I acknowledge i may be very far off in terms of feasible ideas about this. Please excuse my ignorance.

Answer 1

The reason .so (or .pyd) files take up memory space only once (except for their variables segment) is that they are recognized by the OS kernel as object code. .py files are only recognized as text files/data; it's the Python interpreter that grants them "code" status. Embedding the Python interpreter in a shared library won't resolve this.

Loading .py files only once despite their use in multiple processes would require changes deep inside CPython.

Your best option, if you want to save memory space, is to compile Python modules to .so files using Cython. That may require some changes to the modules.