How to make a module reload in python after the script is compiled?

Question

The basic idea involved:

I am trying to make an application where students can write code related to a specific problem(say to check if the number is even) The code given by the student is then checked by the application by comparing the output given by the user's code with the correct output given by the correct code which is already present in the application.

The basic version of the project I am working on:

An application in which you can write a python script (in tkinter text box). The contents of the text box are first stored in a test_it.py file. This file is then imported (on the click of a button) by the application. The function present in test_it.py is then called to get the output of the code(by the user).

The problem:

Since I am "importing" the contents of test_it.py , therefore, during the runtime of the application the user can test his script only once. The reason is that python will import the test_it.py file only once. So even after saving the new script of the user in test_it.py , it wont be available to the application.

The solution:

Reload test_it.py every time when the button to test the script is clicked.

The actual problem:

While this works perfectly when I run the application from the script, this method fails to work for the compiled/executable version(.exe) of the file (which is expected since during compilation all the imported modules would be compiled too and so modifying them later will not work)

The question:

I want my test_it.py file to be reloaded even after compiling the application.

---

If you would like to see the working version of the application to test it yourself. You will find it here.

Answer 1

Problem summary:

A test_it.py program is running and has a predicate available, e.g. is_odd(). Every few minutes, a newly written file containing a revised is_odd() predicate becomes available, and test_it wishes to feed a test vector to revised predicate.

There are several straightforward solutions.

1. Don't load the predicate in the current process at all. Serialize the test vector, send it to a newly forked child which computes and serializes results, and examine those results.
2. Typically eval is evil, but here you might want that, or exec.
3. Replace current process with a newly initialized interpreter: https://docs.python.org/3/library/os.html#os.execl
4. Go the memory leak route. Use a counter to assign each new file a unique module name, manipulate source file to match, and load that. As a bonus, this makes it easy to diff current results against previous results.
5. Reload: from importlib import reload

Answer 2

Even for the bundled application imports work the standard way. That means whenever an import is encountered, the interpreter will try to find the corresponding module. You can make your test_it.py module discoverable by appending the containing directory to sys.path. The import test_it should be dynamic, e.g. inside a function, so that it won't be discovered by PyInstaller (so that PyInstaller won't make an attempt to bundle it with the application).

Consider the following example script, where the app data is stored inside a temporary directory which hosts the test_it.py module:

import importlib
import os
import sys
import tempfile

def main():
    with tempfile.TemporaryDirectory() as td:
        f_name = os.path.join(td, 'test_it.py')

        with open(f_name, 'w') as fh:  # write the code
            fh.write('foo = 1')

        sys.path.append(td)  # make available for import
        import test_it
        print(f'{test_it.foo=}')

        with open(f_name, 'w') as fh:  # update the code
            fh.write('foo = 2')

        importlib.reload(test_it)
        print(f'{test_it.foo=}')

main()

Answer 3

The key is to check if program are running as exe and add exe path to the sys.path.

File program.py:

import time
import sys
import os
import msvcrt
import importlib

if getattr(sys, 'frozen', False):
    # This is .exe so we change current working dir
    # to the exe file directory:
    app_path = os.path.dirname(sys.executable)
    print('    Add .exe path to sys.path: ' + app_path)
    sys.path.append(app_path)
    os.chdir(app_path)

test_it = importlib.import_module('test_it')

def main():
    global test_it
    try:
        print('    Start')
        while True:
            if not msvcrt.kbhit(): continue
            key = msvcrt.getch()
            if key in b'rR':
                print('    Reload module')
                del sys.modules['test_it']
                del test_it
                test_it = importlib.import_module('test_it')
            elif key in b'tT':
                print('    Run test')
                test_it.test_func()
            time.sleep(0.001)
    except KeyboardInterrupt:
        print('    Exit')

if __name__ == '__main__': main()

File test_it.py:

def test_func():
    print('Hi')

Create an .exe file:

pyinstaller --onefile  --clean program.py

Copy _text_it.py to the _dist_ folder and it's done. Press t in program window to run test_func. Edit test_it.py then press r to reload module and press t again to see changes.

Answer 4

Maybe the solution is to use the code module:

import code
# get source from file as a string
src_code = ''.join(open('test_it.py').readlines())
# compile the source
compiled_code = code.compile_command(source=src_code, symbol='exec')
# run the code
eval(compiled_code) # or exec(compiled_code)