Dynamically loading python source code

Question

I'm currently playing with Flask and I can't figure out how the debug mechanism is working. To be more precise when I save the python file with my application I don't need to restart the server, it will be loaded automatically when I make a request. So my question is how does the running program know it was changed and respond to that changes?

Answer 1

Flask is using Werkzug's underlying run_with_reloader function (found in serving.py) ... which is itself using the restart_with_reloader and reloader_loop function created earlier in the same file.

run_with_reloader spawns another python process (running Werkzug again with all the same arguments that you passed to the first one) and this new processes uses the thread module to spawn a new thread or subprocess that runs your server function. It then runs the reloader_loop and waits.

reloader_loop simply loops through all the modules that have been imported and gets their last modified dates. Then at specified intervals (which defaults to 1 s) it checks all the files again to see if they've been modified. If they have, the currently running (slave) Werkzug process exits (terminates) with an exit code of 3. Once it exits, the thread or subprocess it started (which is actually doing the work) is terminated as well. The master process checks to see if the exit code was 3. If it was, it spawns a new slave subprocess, just as it did before. Otherwise, it exits with the same exit code.

Here's the code for reference:

def reloader_loop(extra_files=None, interval=1):
    """When this function is run from the main thread, it will force other
    threads to exit when any modules currently loaded change.

    Copyright notice.  This function is based on the autoreload.py from
    the CherryPy trac which originated from WSGIKit which is now dead.

    :param extra_files: a list of additional files it should watch.
    """
    def iter_module_files():
        for module in sys.modules.values():
            filename = getattr(module, '__file__', None)
            if filename:
                old = None
                while not os.path.isfile(filename):
                    old = filename
                    filename = os.path.dirname(filename)
                    if filename == old:
                        break
                else:
                    if filename[-4:] in ('.pyc', '.pyo'):
                        filename = filename[:-1]
                    yield filename

    mtimes = {}
    while 1:
        for filename in chain(iter_module_files(), extra_files or ()):
            try:
                mtime = os.stat(filename).st_mtime
            except OSError:
                continue

            old_time = mtimes.get(filename)
            if old_time is None:
                mtimes[filename] = mtime
                continue
            elif mtime > old_time:
                _log('info', ' * Detected change in %r, reloading' % filename)
                sys.exit(3)
        time.sleep(interval)


def restart_with_reloader():
    """Spawn a new Python interpreter with the same arguments as this one,
    but running the reloader thread.
    """
    while 1:
        _log('info', ' * Restarting with reloader...')
        args = [sys.executable] + sys.argv
        new_environ = os.environ.copy()
        new_environ['WERKZEUG_RUN_MAIN'] = 'true'

        # a weird bug on windows. sometimes unicode strings end up in the
        # environment and subprocess.call does not like this, encode them
        # to latin1 and continue.
        if os.name == 'nt':
            for key, value in new_environ.iteritems():
                if isinstance(value, unicode):
                    new_environ[key] = value.encode('iso-8859-1')

        exit_code = subprocess.call(args, env=new_environ)
        if exit_code != 3:
            return exit_code


def run_with_reloader(main_func, extra_files=None, interval=1):
    """Run the given function in an independent python interpreter."""
    if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
        thread.start_new_thread(main_func, ())
        try:
            reloader_loop(extra_files, interval)
        except KeyboardInterrupt:
            return
    try:
        sys.exit(restart_with_reloader())
    except KeyboardInterrupt:
        pass