Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does this lambda/yield/generator comprehension work?

I was looking through my codebase today and found this:

def optionsToArgs(options, separator='='):     kvs = [         (             "%(option)s%(separator)s%(value)s" %               {'option' : str(k), 'separator' : separator, 'value' : str(v)}         ) for k, v in options.items()     ]     return list(         reversed(             list(                     (lambda l, t:                          (lambda f:                              (f((yield x)) for x in l)                         )(lambda _: t)                     )(kvs, '-o')                 )             )         ) 

It seems to take a dict of parameters and turn them into a list of parameters for a shell command. It looks like it's using yield inside a generator comprehension, which I thought would be impossible...?

>>> optionsToArgs({"x":1,"y":2,"z":3}) ['-o', 'z=3', '-o', 'x=1', '-o', 'y=2'] 

How does it work?

like image 911
Dog Avatar asked Apr 11 '13 18:04

Dog


People also ask

What is a generator comprehension?

A generator comprehension is a single-line specification for defining a generator in Python. It is absolutely essential to learn this syntax in order to write simple and readable code. Note: Generator comprehensions are not the only method for defining generators in Python.

Can a lambda yield?

While you can technically put a yield in a lambda function, the constraints of lambda functions make it essentially never a useful thing to do.

How does yield work in Python?

What Is Yield In Python? The Yield keyword in Python is similar to a return statement used for returning values or objects in Python. However, there is a slight difference. The yield statement returns a generator object to the one who calls the function which contains yield, instead of simply returning a value.

What is the difference between a generator and comprehension?

So what's the difference between Generator Expressions and List Comprehensions? The generator yields one item at a time and generates item only when in demand. Whereas, in a list comprehension, Python reserves memory for the whole list. Thus we can say that the generator expressions are memory efficient than the lists.


2 Answers

Since Python 2.5, yield <value> is an expression, not a statement. See PEP 342.

The code is hideously and unnecessarily ugly, but it's legal. Its central trick is using f((yield x)) inside the generator expression. Here's a simpler example of how this works:

>>> def f(val): ...     return "Hi" >>> x = [1, 2, 3] >>> list(f((yield a)) for a in x) [1, 'Hi', 2, 'Hi', 3, 'Hi'] 

Basically, using yield in the generator expression causes it to produce two values for every value in the source iterable. As the generator expression iterates over the list of strings, on each iteration, the yield x first yields a string from the list. The target expression of the genexp is f((yield x)), so for every value in the list, the "result" of the generator expression is the value of f((yield x)). But f just ignores its argument and always returns the option string "-o". So on every step through the generator, it yields first the key-value string (e.g., "x=1"), then "-o". The outer list(reversed(list(...))) just makes a list out of this generator and then reverses it so that the "-o"s will come before each option instead of after.

However, there is no reason to do it this way. There are a number of much more readable alternatives. Perhaps the most explicit is simply:

kvs = [...] # same list comprehension can be used for this part result = [] for keyval in kvs:    result.append("-o")    result.append(keyval) return result 

Even if you like terse, "clever" code, you could still just do

return sum([["-o", keyval] for keyval in kvs], []) 

The kvs list comprehension itself is a bizarre mix of attempted readability and unreadability. It is more simply written:

kvs = [str(optName) + separator + str(optValue) for optName, optValue in options.items()] 

You should consider arranging an "intervention" for whoever put this in your codebase.

like image 191
BrenBarn Avatar answered Sep 20 '22 23:09

BrenBarn


Oh god. Basically, it boils down to this,:

def f(_):              # I'm the lambda _: t     return '-o'  def thegenerator():   # I'm (f((yield x)) for x in l)     for x in kvs:         yield f((yield x)) 

So when iterated over, thegenerator yields x (a member of kvs) and then the return value of f, which is always -o, all in one iteration over kvs. Whatever yield x returns and what gets passed to f is ignored.

Equivalents:

def thegenerator():   # I'm (f((yield x)) for x in l)     for x in kvs:         whatever = (yield x)         yield f(whatever)  def thegenerator():   # I'm (f((yield x)) for x in l)     for x in kvs:         yield x         yield f(None)  def thegenerator():   # I'm (f((yield x)) for x in l)     for x in kvs:         yield x         yield '-o' 

There are lots of ways to do this much simpler, of course. Even with the original double-yield trick, the entire thing could've been

return list(((lambda _: '-o')((yield x)) for x in kvs))[::-1] 
like image 20
Pavel Anossov Avatar answered Sep 20 '22 23:09

Pavel Anossov