LINQ/IEnumerable Skip().Take() efficiency with used with "yield return"

Question

I have a question about the efficiency of Skip() and Take() when used with IEnumerable<>.

I am returning all my data lists with IEnumerable<> and i use 'yield return' to prevent me from having to allocate large amounts of memory to pass back the data. This works very efficiently.

However, later in my process I wanted to batch this data and take blocks of say 20 entries from my list at a time. I thought to myself.. ah! This fits an enumerator perfectly.

I discovered the very useful Skip() and Take() methods on the IEnumerable interface however I'm now realising that this causes my Loop to re-interate from the beginning each time.

What is the best way of paging data from an IEnumerable? Am I better off using MoveFirst() and MoveNext() on the enumerator instead of Skip() and Take()?

I've done some googling but can't find an answer..

Can anyone help?

I really love LINQ functionality on IEnumerable<> however I really have to take efficiency into consideration.

Answer 1

You can write a Batch method to transform a sequence of items into a sequence of batches of a given size, which can be done without needing to iterate the source sequence multiple times, and which can limit the memory footprint to only holding the size of one batch in memory at once:

public static IEnumerable<IEnumerable<T>> Batch<T>(
    this IEnumerable<T> source, int batchSize)
{
    List<T> buffer = new List<T>(batchSize);

    foreach (T item in source)
    {
        buffer.Add(item);

        if (buffer.Count >= batchSize)
        {
            yield return buffer;
            buffer = new List<T>(batchSize);
        }
    }
    if (buffer.Count > 0)
    {
        yield return buffer;
    }
}