asp.net Core 1.1 chunked responses

Question

Hopefully, someone can enlighten me on the following.

I have a client that will do a request to a controller endpoint (there is no view, c# to c# or even C++ later). That controller will have to send responses as it fetches them asynchronously as json (sends json1 to client, then json2, then json3 until it closes the connection or send a null terminated text or similar). The purpose is to stream the results back to the client so it can start processing while the server still works.

My controller endpoint looks like this:

        [HttpGet("testStream")]
        public async Task testStream()
        {
            var response = HttpContext.Response;
            response.Headers[HeaderNames.TransferEncoding] = "chunked";


            for (var i = 0; i < 10; ++i)
            {
                await response.WriteAsync($"6\r\ntest {i}\r\n");

                await response.Body.FlushAsync();
                await Task.Delay(1 * 1000);
            }
            await response.WriteAsync("0\r\n\r\n");
            await response.Body.FlushAsync();
        }

My test looks like this:

        static async void DownloadPageAsync()
        {
            // ... Target page.
            string page = "http://localhost:8080/api/Stream/testStream";
            Console.WriteLine("test");
            while (!Debugger.IsAttached) Thread.Sleep(500);
            // ... Use HttpClient.
            using (HttpClient client = new HttpClient())
            using (var response = await client.GetAsync(page, HttpCompletionOption.ResponseHeadersRead))
            using (HttpContent content = response.Content)
            {
                string result = await content.ReadAsStringAsync();
                do
                {
                    Console.WriteLine(result);
                    result = await content.ReadAsStringAsync();
                }
                while (result != "null");
            }
            Console.WriteLine("END");
        }
        [Fact]
        public void Test1()
        {
            TestSurvey.DownloadPageAsync();
        }

I am getting exception when I call content.ReadAsStringAsync();

System.Net.Http.HttpRequestException : Error while copying content to a stream.
[xUnit.net 00:01:14.5836121]       ---- System.IO.IOException : The read operation failed, see inner exception.
[xUnit.net 00:01:14.5836496]       -------- System.Net.Http.CurlException : Failure when receiving data from the peer
[xUnit.net 00:01:14.5846837]       Stack Trace:
[xUnit.net 00:01:14.5857807]            at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__48.MoveNext()

EDIT: Exception was due to not sending the size of the chunk await response.WriteAsync($"6\r\ntest {i}\r\n");

but now on the test/client side, I get all the chunks at once...

Answer 1

To solve this, I made use of SSE or Server Side Events. here is the server side in asp.net core:

        [HttpGet("testStream")]
        public async Task testStream()
        {
            var response = HttpContext.Response;
            response.StatusCode = 200;
            response.ContentType = "text/event-stream";

            for (var i = 0; i < 10; ++i)
            {
                //the tags are either 'events:' or 'data:' and two \n indicates ends of the msg
                //event: xyz \n\n
                //data: xyz \n\n
                await response.WriteAsync($"data: test {i}\n\n");

                response.Body.Flush();
                await Task.Delay(5 * 1000);
            }
            await response.WriteAsync("data:\n\n");
            await response.Body.FlushAsync();
        }

and here is the client side:

        string page = "http://localhost:8080/api/Stream/testStream";

        //while (!Debugger.IsAttached) Thread.Sleep(500);

        using (HttpClient client = new HttpClient())
        using (var s = await client.GetStreamAsync(page))
        {
            using (StreamReader r = new StreamReader(s))
            {
                string line = null;
                while (null != (line = r.ReadLine()))
                {
                    Console.WriteLine(line);
                }
            }
        }

Usage of ReadAsStringAsync forced wait of all the message in order to proceed.