C# Stream Response from 3rd party, minimal buffering

Question

Our ASP.NET MVC endpoint is a behaving as a proxy to another 3rd party HTTP endpoint, which returns about 400MB of XML document generated dynamically.

Is there a way for ASP.NET MVC to "stream" that 3rd party response straight to the user of our endpoint with "minimal" buffering ?

At the moment, it looks like ASP.NET System.Web.Mvc.Controller.File() loads the whole file into memory as the response. Not sure how I can confirm this, other than the jump in memory usage ?

System.Web.Mvc.Controller.File(

The IIS AppPool memory usage increases by 400MB, which is then re-claimed by Garbage Collection later.

It will be nice if we can avoid System.Web.Mvc.Controller.File() loading the whole 400MB strings into memory, by streaming it "almost directly" from incoming response, is it possible ?

The mock c# linqpad code is roughly like this

public class MyResponseItem {
    public Stream myStream;
    public string metadata;
}

void Main()
{
    Stream stream = MyEndPoint();   

    //now let user download this XML as System.Web.Mvc.FileResult
    System.Web.Mvc.ActionResult fileResult = System.Web.Mvc.Controller.File(stream, "text/xml");
    fileResult.Dump();
}

Stream MyEndPoint() {
    MyResponseItem myResponse = GetStreamFromThirdParty("https://www.google.com");
    return myResponse.myStream;
}

MyResponseItem GetStreamFromThirdParty(string fullUrl)
{   
    MyResponseItem myResponse = new MyResponseItem();   
    System.Net.WebResponse webResponse = System.Net.WebRequest.Create(fullUrl).GetResponse();
    myResponse.myStream = webResponse.GetResponseStream();
    return myResponse;
}

Answer 1

You can reduce the memory footprint by not buffering and just copying the stream directly to output stream, an quick n' dirty example of this here:

    public async Task<ActionResult> Download()
    {
        using (var httpClient = new System.Net.Http.HttpClient())
        {
            using (
                var stream = await httpClient.GetStreamAsync(
                    "https://ckannet-storage.commondatastorage.googleapis.com/2012-10-22T184507/aft4.tsv.gz"
                    ))
            {
                Response.ContentType = "application/octet-stream";
                Response.Buffer = false;
                Response.BufferOutput = false;
                await stream.CopyToAsync(Response.OutputStream);
            }
            return new HttpStatusCodeResult(200);
        }
    }

If you want to reduce the footprint even more you can set a lower buffer size with the CopyToAsync(Stream, Int32) overload, default is 81920 bytes.