Idioms in python: closure vs functor vs object

Question

So I am curious about the views of more experienced python programmers on the following style question. Suppose that I am building a function that is going to iterate row by row through a pandas dataframe or any similar use-case where a function requires access to its previous state. There seem to be at least four ways to implement this in python:

1. Closures:

def outer():
    previous_state = None
    def inner(current_state) :
        nonlocal previous_state
        #do something
        previous_state=current_state
        return something

So if you come from a javascript background this will doubtless seem natural to you. It feels pretty natural in python too, right up until you need to access the enclosing scope when you will end up doing something like inner.__code__.co_freevars, which will give you the names of your enclosing variables as a tuple, and finding the index of the one you want, and then going to inner.__closure__[index].cell_contents to get its value. Not exactly elegant, but I suppose the point is often to hide scope, so it makes sense that it should be hard to reach. On the other hand, it also feels a bit weird that python makes the enclosing function private when it has done away with almost every other way to have a private variable compared to OOP languages.

2. Functor

def outer():
    def inner(current_state):
        #do something
        inner.previous_state=current_state
        return something
    ret = inner
    ret.previous_state=None
    return ret

This "opens the closure" in that now the enclosing state is fully visible as an attribute of the function. This works because functions are really just objects in disguise. I am leaning towards this as the most pythonic. Its clear, concise, and readable.

3. Objects This is probably the most familiar to OOP programmers

class Calculator(Object) :
    def __init__(self):
        self.previous_state=None

    def do_something(self, current_state) :
        #do_something
        self.previous_state = current_state
        return something

The biggest con here is that you tend to end up with a lot of class definitions. That is fine in a fully OOP language like Java where you have interfaces and the like to manage this, but it seems a bit odd in a duck typed language to have many simple classes just to carry around a function that needs a bit of state.

4. globals - I won't demonstrate this as I specifically want to avoid polluting the global namespace

5. Decorators - this is a little bit of a curveball, but you can use decorators to store partial state information.

@outer
def inner(previous_state, current_state):
    #do something
    return something

def outer(inner) :
    def wrapper(current_state) :
        result =  inner(wrapper.previous_state, current_state)
        wrapper.previous_state = current_state
        return result
    ret = wrapper
    ret.previous_state=None
    return result

This kind of syntax is the least familiar to me, but if I now call

func = inner

I actually get

func = outer(inner)

and then repeatedly calling func() acts just like the functor example. I actually really hate this way the most. It seems to me to have a really non transparent syntax in that it isn't clear if calling inner(current_state) lots of times will give you the same result or if it will give you a newly decorated function every time, so it seems like bad practice to make decorators which add state to a function in this way.

So which is the correct way? What pros and cons have I missed here?

Answer 1

So the correct answer to this is the callable object, which essentially replaces the idiom of the closure in python.

so working off option 3 above change:

class Calculator(Object) :
    def __init__(self):
        self.previous_state=None

    def do_something(self, current_state) :
        #do_something
        self.previous_state = current_state
        return something

to

class Calculator(Object) :
    def __init__(self):
        self.previous_state=None

    def __call__(self, current_state) :
        #do_something
        self.previous_state = current_state
        return something

and now you can call it like a function. So

func = Calculator():
for x in list:
    func(x)

Answer 2

You can define a generator, which is a restricted form of a coprocess.

def make_gen():
    previous_state = None
    for row in rows:
        # do something
        previous_state = current_state
        yield something

thing = make_gen()
for item in thing:
    # Each iteration, item is a different value
    # "returned" by the yield statement in the generator

Instead of calling thing (which replaces your inner function) repeatedly, you iterate over it (which is basically the same as calling next(thing) repeatedly).

The state is entirely contained within the body of the generator.

If you don't want to actually iterate over it, you can still selectively "re-enter" the coprocess by calling next explicitly.

thing = make_gen()
first_item = next(thing)
# do some stuff
second_item = next(thing)
# do more stuff
third_item = next(thing)
fourth_item = next(thing)
# etc