How does Python's types.FunctionType create dynamic Functions?

Question

I'm trying to strengthen my Python skills, and I came across Open-Source code for Saltstack that is using types.FunctionType, and I don't understand what's going on.

salt.cloud.clouds.cloudstack.py

Function create() has the following bit of code:

kwargs = {
    'name': vm_['name'],
    'image': get_image(conn, vm_),
    'size': get_size(conn, vm_),
    'location': get_location(conn, vm_),
}

The function get_image, and get_size are passed to a function 'namespaced_function' as so:

get_size = namespaced_function(get_size, globals())
get_image = namespaced_function(get_image, globals())

salt.utils.functools.py

Has the namespaced function

def namespaced_function(function, global_dict, defaults=None, preserve_context=False):
    '''
    Redefine (clone) a function under a different globals() namespace scope
        preserve_context:
            Allow keeping the context taken from orignal namespace,
            and extend it with globals() taken from
            new targetted namespace.
    '''
    if defaults is None:
        defaults = function.__defaults__

    if preserve_context:
        _global_dict = function.__globals__.copy()
        _global_dict.update(global_dict)
        global_dict = _global_dict
    new_namespaced_function = types.FunctionType(
        function.__code__,
        global_dict,
        name=function.__name__,
        argdefs=defaults,
        closure=function.__closure__
    )
    new_namespaced_function.__dict__.update(function.__dict__)
    return new_namespaced_function

I can see that they are dynamically creating a function get_image, but I don't understand the benefit of doing it this way. Why not just create the function?

Answer 1

Since namespaced_function() takes the (old) function get_image() as an argument and returns the (new) get_image() function, it's more apt to say that it's modifying the function, rather than creating it. Of course, it is creating a function and returning it, but one that very closely resembles the input function. namespaced_function() works a bit like a decorator, but decorators usually just wrap the whole input function inside another function that calls the original, rather than actually creating a modified version of the original function. The original get_image() is defined at libcloudfuncs.py.

So the question becomes, "How does namespaced_function() modify the input function?". If you look at what types.FunctionType() is getting as its arguments, you see that most values get copied over directly from the original function. The only argument that isn't directly copied is the function's globals. In other words, namespaced_function() is creating a new function, that is identical in every way to the input function, except that when the function refers to global variables, it looks for them in a different place.

So, they're creating a new version of get_image() that also has access to the current module's global variables. Why are they doing that? Well, either to override some global variables, or to provide ones that weren't there at all in the original module (in which case the original function would have been deliberately broken before the modification). But I can't really answer the "Why?" except by summarily saying that they probably judged that it was easier than the alternatives.

So what are the alternatives? Well, global variables are often frowned upon - when they aren't constant - because, well, you might want to change them. They could have used extra arguments instead of global variables, but probably didn't want to have to keep passing the same arguments around, when most their functions use them. You can inject arguments too, though, like they injected the globals - and it's less hacky, too! So why didn't they do that? Well, again, I kind of have to guess, but they probably have more than one global variable they're changing/providing.

Automatically providing arguments is easy:

def original(auto1, arg1, arg2, auto2):
    print(auto1, auto2, arg1, arg2)

injected = functools.partial(original, 'auto1', auto2='auto2')

injected(1, 2) # is equal to original('auto1', 1, 2, 'auto2')

Automatically providing lots of arguments gets tedious quite quickly.

Of course, you might just make all the functions have an argument named eg. globals as the first argument, and use injected = functools.partial(original, globals()). But then inside the function, whenever you needed to refer to such a variable, you'd need to say globals['variable'] instead of just variable.

So, in conclusion, it's perhaps a bit hacky, but the authors have probably judged that "a bit hacky" is still a lot better than the more verbose alternatives.