How to concat async enumerables?

Question

I have a method with this return type:

public async Task<IEnumerable<T>> GetAll()

It makes some further async calls (unknown number) each of which return a task of enumerable T, and then wants to concat the results for the return.

var data1 = src1.GetAll();
var data2 = src2.GetAll();
var data3 = src3.GetAll(); //and so on

Now it's easy enough to await all and concat the results to produce the single enumerable, but i'd like the enumerable to be available as soon as the first call returns, with potential waits for the caller / enumerator if any calls are still pending when available results run out.

Do i have to hand-roll a concat for this, working around the lack of enumerator support when it's wrapped in a task<>? Or there's already a library call in TPL or elsewhere which could help me. I did look at IX, but it's still on experimental release and don't want to fold it in.

On a side note, is what i'm trying an anti-pattern? I can think of one complication, exception handling - from the caller's side, the call can complete successfully and he starts using the enumerable but it can blow up midway through that...

Answer 1

There is an existing project called Async Enumerable which answers this problem exactly.

You could put it to use quite easily.

For example:

IAsyncEnumerable<string> GetAsyncAnswers()
{
    return AsyncEnum.Enumerate<string>(async consumer =>
    {
        foreach (var question in GetQuestions())
        {
            string theAnswer = await answeringService.GetAnswer(question);
            await consumer.YieldAsync(theAnswer);
        }
    });
}

This exposes an IAsyncEnumerable<string> which yields once GetAnswer returns. You could internally expose an IAsyncEnumerable<T> in your case and internally make calls inside GetAll.

what i'm trying an anti-pattern? I can think of one complication, exception handling - from the caller's side, the call can complete successfully and he starts using the enumerable but it can blow up midway through that...

I wouldn't say so. This does have potentially problems such as an exception occurring internally during one of the awaits, but this could also potentially happen inside any IEnumerable<T>. Asynchronous sequences are something needed in todays reality of emerging async API's.