Python `yield from`, or return a generator?

Question

I wrote this simple piece of code:

def mymap(func, *seq):   return (func(*args) for args in zip(*seq)) 

Should I use the 'return' statement as above to return a generator, or use a 'yield from' instruction like this:

def mymap(func, *seq):   yield from (func(*args) for args in zip(*seq)) 

and beyond the technical difference between 'return' and 'yield from', which is the better approach the in general case?

Answer 1

The difference is that your first mymap is just a usual function, in this case a factory which returns a generator. Everything inside the body gets executed as soon as you call the function.

def gen_factory(func, seq):     """Generator factory returning a generator."""     # do stuff ... immediately when factory gets called     print("build generator & return")     return (func(*args) for args in seq) 

The second mymap is also a factory, but it's also a generator itself, yielding from a self-built sub-generator inside. Because it is a generator itself, execution of the body does not start until the first invokation of next(generator).

def gen_generator(func, seq):     """Generator yielding from sub-generator inside."""     # do stuff ... first time when 'next' gets called     print("build generator & yield")     yield from (func(*args) for args in seq) 

I think the following example will make it clearer. We define data packages which shall be processed with functions, bundled up in jobs we pass to the generators.

def add(a, b):     return a + b  def sqrt(a):     return a ** 0.5  data1 = [*zip(range(1, 5))]  # [(1,), (2,), (3,), (4,)] data2 = [(2, 1), (3, 1), (4, 1), (5, 1)]  job1 = (sqrt, data1) job2 = (add, data2) 

Now we run the following code inside an interactive shell like IPython to see the different behavior. gen_factory immediately prints out, while gen_generator only does so after next() being called.

gen_fac = gen_factory(*job1) # build generator & return <-- printed immediately next(gen_fac)  # start # Out: 1.0 [*gen_fac]  # deplete rest of generator # Out: [1.4142135623730951, 1.7320508075688772, 2.0]  gen_gen = gen_generator(*job1) next(gen_gen)  # start # build generator & yield <-- printed with first next() # Out: 1.0 [*gen_gen]  # deplete rest of generator # Out: [1.4142135623730951, 1.7320508075688772, 2.0] 

To give you a more reasonable use case example for a construct like gen_generator we'll extend it a little and make a coroutine out of it by assigning yield to variables, so we can inject jobs into the running generator with send().

Additionally we create a helper function which will run all tasks inside a job and ask as for a new one upon completion.

def gen_coroutine():     """Generator coroutine yielding from sub-generator inside."""     # do stuff... first time when 'next' gets called     print("receive job, build generator & yield, loop")     while True:         try:             func, seq = yield "send me work ... or I quit with next next()"         except TypeError:             return "no job left"         else:             yield from (func(*args) for args in seq)   def do_job(gen, job):     """Run all tasks in job."""     print(gen.send(job))     while True:         result = next(gen)         print(result)         if result == "send me work ... or I quit with next next()":             break 

Now we run gen_coroutinewith our helper function do_joband two jobs.

gen_co = gen_coroutine() next(gen_co)  # start # receive job, build generator & yield, loop  <-- printed with first next() # Out:'send me work ... or I quit with next next()' do_job(gen_co, job1)  # prints out all results from job # 1 # 1.4142135623730951 # 1.7320508075688772 # 2.0 # send me work... or I quit with next next() do_job(gen_co, job2)  # send another job into generator # 3 # 4 # 5 # 6 # send me work... or I quit with next next() next(gen_co) # Traceback ... # StopIteration: no job left 

To come back to your question which version is the better approach in general. IMO something like gen_factory makes only sense if you need the same thing done for multiple generators you are going to create, or in cases your construction process for generators is complicated enough to justify use of a factory instead of building individual generators in place with a generator comprehension.

Note:

The description above for the gen_generator function (second mymap) states "it is a generator itself". That is a bit vague and technically not really correct, but facilitates reasoning about the differences of the functions in this tricky setup where gen_factory also returns a generator, namely that one built by the generator comprehension inside.

In fact any function (not only those from this question with generator comprehensions inside!) with a yield inside, upon invocation, just returns a generator object which gets constructed out of the function body.

type(gen_coroutine) # function
gen_co = gen_coroutine(); type(gen_co) # generator

So the whole action we observed above for gen_generator and gen_coroutine takes place within these generator objects, functions with yield inside have spit out before.