When do a C# Webclient.OpenRead actually download data?

Question

I found this answer for Python. Does it apply to C# WebClient.OpenRead?

In the following example:

1. Does OpenRead downloads all the csv file at once (hence ReadLine refers to a local Stream)?
2. Like in Python, does the download is being done progressively with successive ReadLine?

Code sample

WebClient client = new WebClient();
Stream stream = client.OpenRead("http://www.MyWebsite.com/FileToDownload.csv");
StreamReader csvFile= new StreamReader(stream);
while (!csvFile.EndOfStream)
{
    string line = csvFile.ReadLine();
    //do stuff with line
}

Answer 1

It depends on the protocol, size of the download vs. buffer size and may even depend on the web server configuration. The implementation of HttpWebResponse, which is used internally, reads everything from a ConnectStream, which is capable of de-chunking transparently:

ConnectStream in reference source

Even in cases where decoding or decompressing is needed, streams are properly chained, so the implementation is clearly able to stream data without downloading everything locally up front.
But in practice the .NET http stack buffers as much as it can immediately, also when you're not reading a stream to end. These posts may be an interesting read:

• https://weblog.west-wind.com/posts/2014/jan/29/using-net-httpclient-to-capture-partial-responses
• How to retrieve partial response with System.Net.HttpClient

To prove for your case, turn on Network Tracing, as HttpWebResponse will then dump information about connects, continuations, etc. Or use any other network sniffing tool. Be sure the file is large enough to see the effect.

Strictly speaking it will never be a "local stream", but the underlying buffer will hold more, if not all, of the requested file than you ask for in the first read.