How do I load a Python module from a string while preserving debug?

Question

I'm interested in loading a Python module that has its source embedded in a C extension. It should be possible to do something with Python's importlib machinery like importlib.util.spec_from_file_location so that the source code will appear if you are debugging. How would I implement an importlib.util.spec_from_string?

Answer 1

Here's how to define a loader that takes the module's source from a string, and then creates and loads the module into sys.modules. It could be useful if the module's source is not in a file. If there is already a file then use https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

Although inspect.getsource(module) works for a subclass of importlib.abc.InspectLoader for which it would only be necessary to define get_source, tracebacks and pdb don't appear to be willing to display the source code until you inherit from SourceLoader.

import sys
import importlib.abc, importlib.util

class StringLoader(importlib.abc.SourceLoader):
    def __init__(self, data):
        self.data = data

    def get_source(self, fullname):
        return self.data
    
    def get_data(self, path):
        return self.data.encode("utf-8")
    
    def get_filename(self, fullname):
        return "<not a real path>/" + fullname + ".py"
    
module_name = "testmodule"
with open("testmodule.py", "r") as module:
    loader = StringLoader(module.read())

spec = importlib.util.spec_from_loader(module_name, loader, origin="built-in")
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)

Answer 2

As a quick fix, you can dump it in a temporary module, import it using exec and delete the temp module when you're done.

Here's a toy example:

dummy_src =""" 
print("imported!") 
x = 5 
""" 

with open("temp.py", "w") as f: 
    f.write(dummy_src) 

exec("import temp") 
print(temp.x)

Output:

imported!
5