Cythonizing for loops that iterate over generators

Question

I have Python code that has lots of loops that consume data from Python generators. Some also re yield the processed data. This is a bottleneck and I want to speed this part up and was thinking of using Cython.

What is the recommended way to deal with generators and yield. I would like to

• Convert Python generators into Cython without data copies
• Make Cython for loops consume data produced by Python generators
• Yield data like a generator

I would guess this is a common enough use case, what is the recommended ways to do this.

Answer 1

Some recommendations:

• Cython supports generators out of the box, so you should try passing your Python code with generators to cython and see what kind of speedup you get.
• Next step is to add as much static typing information to your loops to speed up the work the generators are doing.
• Python generators are cool, but if performance is important, they aren't the fastest way to do things. You're much better off converting your bottlenecks to working with contiguous arrays.
• Check out Cython's typed memoryviews.
• You can also use Cython with C++ std::vectors and other high-performance container objects.

We'll need more information about your goals and constraints to provide more help here. A stripped down example would be helpful.