Handling flexible function arguments in Python

Question

TL;TR Looking for idioms and patterns to unpack positional and keyword arguments into ordered sequence of positional arguments, based on simple specification, e.g. a list of names. The idea seems similar to scanf-like parsing.

I'm wrapping functions of a Python module, called someapi. Functions of someapi only expect positional arguments, which are in pain numbers in most cases. I'd like to enable callers with flexibility of how they can pass arguments to my wrappers. Here are examples of the wrappers invocations I'd like to allow:

# foo calls someapi.foo()
foo(1, 2, 3, 4)
foo(1, 2, 3, 4, 5) # but forward only 1st 4 to someapi.foo
foo([1, 2, 3, 4])
foo([1, 2, 3, 4, 5, 6]) # but forward only 1st 4 to someapi.foo
foo({'x':1, 'y':2, 'z':3, 'r':4})
foo(x=1, y=2, z=3, r=4)
foo(a=0, b=0, x=1, y=2, z=3, r=4) # but forward only x,y,z,r someapi.foo

I don't see any need to support convoluted case of mixed positional and keyword arguments:

foo(3, 4, x=1, y=2)

Here is my first stab at implementing such arguments handling for the foo wrapper calling someapi.foo:

def foo(*args, **kwargs):
    # BEGIN arguments un/re-packing
    a = None
    kwa = None
    if len(args) > 1:
        # foo(1, 2, 3, 4)
        a = args
    elif len(args) == 1:
        if isinstance(args[0], (list, tuple)) and len(args[0]) > 1:
            # foo([1, 2, 3, 4])
            a = args[0]
        if isinstance(args[0], dict):
            # foo({'x':1, 'y':2, 'z':3, 'r':4})
            kwa = args[0]
    else:
        # foo(x=1, y=2, z=3, r=4)
        kwa = kwargs

    if a:
        (x, y, z, r) = a
    elif kwa:
        (x, y, z, r) = (kwa['x'], kwa['y'], kwa['z'], kwa['r'])
    else:
        raise ValueError("invalid arguments")
    # END arguments un/re-packing

    # make call forwarding unpacked arguments 
    someapi.foo(x, y, z, r)

It does the job as expected, as far as I can tell, but it there are two issues:

1. Can I do it better in more Python idiomatic fashion?
2. I have dozen(s) of someapi functions to wrap, so how to avoid copying and adjusting the whole block between BEGIN/END marks in every wrapper?

I don't know the answer for the question 1, yet.

Here, however, is my attempt to address the issue 2.

So, I defined a generic handler for arguments based on the simple specification of names. The names specify a couple of things, depending on the actual wrapper invocation:

• How many arguments to unpack from *args? (see len(names) test below)
• What keyword arguments are expected in **kwargs? (see generator expression returning tuple below)

Here is new version:

def unpack_args(names, *args, **kwargs):
    a = None
    kwa = None
    if len(args) >= len(names):
        # foo(1, 2, 3, 4...)
        a = args
    elif len(args) == 1:
        if isinstance(args[0], (list, tuple)) and len(args[0]) >= len(names):
            # foo([1, 2, 3, 4...])
            a = args[0]
        if isinstance(args[0], dict):
            # foo({'x':1, 'y':2, 'z':3, 'r':4...})
            kwa = args[0]
    else:
        # foo(x=1, y=2, z=3, r=4)
        kwa = kwargs
    if a:
        return a
    elif kwa:
        if all(name in kwa.keys() for name in names):
            return (kwa[n] for n in names)
        else:
            raise ValueError("missing keys:", \
                [name for name in names if name not in kwa.keys()])
    else:
        raise ValueError("invalid arguments")

This allows me to implement the wrapper functions in the following way:

def bar(*args, **kwargs):
    # arguments un/re-packing according to given of names
    zargs = unpack_args(('a', 'b', 'c', 'd', 'e', 'f'), *args, **kwargs)
    # make call forwarding unpacked arguments 
    someapi.bar(*zargs)

I think I have achieved all the advantages over the foo version above that I was looking for:

• Enable callers with the requested flexibility.

• Compact form, cut down on copy-and-paste.

• Flexible protocol for positional arguments: bar can be called with 7, 8 and more positional arguments or a long list of numbers, but only first 6 are taken into account. For example, it would allow iterations processing long list of numbers (e.g. think of geometry coordinates):

    # meaw expects 2 numbers
    n = [1,2,3,4,5,6,7,8]
    for i in range(0, len(n), 2):
        meaw(n[i:i+2])

• Flexible protocol for keyword arguments: more keywords may be specified than actually used or dictionary can have more items than used.

Getting back to the question 1 above, can I do better and make it more Pythonic?

Also, I'd like to ask for review of my solution: you see any bugs? have I overlooked anything? how to improve it?

Answer 1

Python is a very powerful language that allows you manipulate code in any way you want, but understanding what you're doing is hard. For this you can use the inspect module. So an example of how to wrap a function in someapi. I'll only consider positional arguments in this example, you can intuit how to extend this further. You can do it like this:

import inspect
import someapi

def foo(args*):
    argspec = inspect.getargspec(someapi.foo)

    if len(args) > len(argspec.args):
        args = args[:len(argspec.args)]

    return someapi.foo(*args)

This will detect if the number of arguments given to foo is too many and if so, it will get rid of the excess arguments. On the other hand, if there are too few arguments then it will just do nothing and let foo handle the errors.

Now to make it more pythonic. The ideal way to wrap many functions using the same template is to use decorator syntax (familiarity with this subject is assumed, if you want to learn more then see the docs at http://www.python.org/doc). Although since decorator syntax is mostly used on functions that are in development rather than wrapping another API, we'll make a decorator but just use it as a factory (the factory pattern) for our API. To make this factory we'll make use of the functools module to help us out (so the wrapped function looks as it should). So we can turn our example into:

import inspect
import functools
import someapi

def my_wrapper_maker(func):
    @functools.wraps(func)
    def wrapper(args*):
        argspec = inspect.getargspec(func)

        if len(args) > len(argspec.args):
            args = args[:len(argspec.args)]

        return func(*args)
    return wrapper

foo = my_wrapper_maker(someapi.foo)

Finally, if someapi has a relatively large API that could change between versions (or we just want to make our source file more modular so it can wrap any API) then we can automate the application of my_wrapper_maker to everything exported by the module someapi. We'll do this like so:

__all__ = ['my_wrapper_maker']

# Add the entire API of someapi to our program.
for func in someapi.__all__:
    # Only add in bindings for functions.
    if callable(getattr(someapi, func)):
        globals()[func] = my_wrapper_maker(getattr(someapi, func))
        __all__.append(func)

This probably considered the most pythonic way to implement this, it makes full use of Python's meta-programming resources and allows the programmer to use this API everywhere they want without depending on a specific someapi.

Note: Whether this is most idiomatic way to do this is really up to opinion. I personally believe that this follows the philosophy set out in "The Zen of Python" quite well and so to me it is very idiomatic.