Module attribute update is not propagated to child processes on Windows

Question

I'm having some issues related to module attribute update on Windows not being propagated to child processes on Windows.

The following snippet illustrates the problem:

import functools
import multiprocessing
import os
from contextlib import contextmanager

_DOMAIN_RANGE_SCALE = 'reference'


def get_domain_range_scale():
    return _DOMAIN_RANGE_SCALE


def set_domain_range_scale(scale='Reference'):
    global _DOMAIN_RANGE_SCALE

    scale = str(scale).lower()

    _DOMAIN_RANGE_SCALE = scale


class domain_range_scale(object):
    def __init__(self, scale):
        self._scale = scale
        self._previous_scale = get_domain_range_scale()

    def __enter__(self):
        set_domain_range_scale(self._scale)

        return self

    def __exit__(self, *args):
        set_domain_range_scale(self._previous_scale)

    def __call__(self, function):
        @functools.wraps(function)
        def wrapper(*args, **kwargs):
            with self:
                return function(*args, **kwargs)

        return wrapper


@contextmanager
def multiprocessing_pool(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)

    yield pool

    pool.terminate()


def test_domain_range_scale(*args):
    print('Domain Range Scale Inner: {0}, PID: {1}'.format(
        get_domain_range_scale(), os.getpid()))


if __name__ == '__main__':
    for scale in ('reference', '1', '100'):
        with domain_range_scale(scale):
            print('*' * 79)
            print('Domain Range Scale Outer: {0}, PID: {1}'.format(
                get_domain_range_scale(), os.getpid()))
            with multiprocessing_pool(processes=4) as pool:
                pool.map(test_domain_range_scale, range(10))

Output on Linux / macOS

*******************************************************************************
Domain Range Scale Outer: reference, PID: 93989
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93992
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93992
*******************************************************************************
Domain Range Scale Outer: 1, PID: 93989
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93995
*******************************************************************************
Domain Range Scale Outer: 100, PID: 93989
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999

Output on Windows

*******************************************************************************
Domain Range Scale Outer: reference, PID: 6524
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 5476
Domain Range Scale Inner: reference, PID: 4872
Domain Range Scale Inner: reference, PID: 1932
*******************************************************************************
Domain Range Scale Outer: 1, PID: 6524
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 1012
Domain Range Scale Inner: reference, PID: 1852
Domain Range Scale Inner: reference, PID: 6544
*******************************************************************************
Domain Range Scale Outer: 100, PID: 6524
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 5944

Answer 1

Your issue lies in Windows not supporting "fork" as start-method for new processes (only "spawn"). Globals are not inherited with "spawn". When you put a print-statement below _DOMAIN_RANGE_SCALE = 'reference', you will see that child processes on Windows will run the script again until if __name__ == '__main__': when they import needed functions.

You will have to use Pool's initializer-parameter to register globals explicitly after process-start.

...

def init_global(scale):
    global _DOMAIN_RANGE_SCALE
    _DOMAIN_RANGE_SCALE = scale

if __name__ == '__main__':

...
        with multiprocessing.Pool(processes=4,
                                  initializer=init_global,
                                  initargs=(scale,)) as pool:

            pool.map(test_domain_range_scale, range(10))
...