Copying from one stream to another?

Question

For work, the specification on my project is to use .Net 2.0 so I don't get the handy CopyTo function brought about later on.

I need to copy the response stream from an HttpWebResponse to another stream (most likely a MemoryStream, but it could be any subclass of Stream). My normal tactic has been something along the lines of:

BufferedStream bufferedresponse = new BufferedStream(HttpResponse.GetResponseStream());
int count = 0;
byte[] buffer = new byte[1024];
do {
    count = bufferedresponse.Read(buffer, 0, buffer.Length);
    target.Write(buffer, 0, count);
} while (count > 0);
bufferedresponse.Close();

Are there more efficient ways to do this? Does the size of the buffer really matter? What is the best way to copy from one stream to another in .Net 2.0?

P.S. This is for downloading large, 200+ MB GIS tif images. Of course reliability is paramount.

Answer 1

This is a handy function. And yes, the buffer size matters. Increasing it might give you better performance on large files.

public static void WriteTo(Stream sourceStream, Stream targetStream)
{
       byte[] buffer = new byte[0x10000];
       int n;
       while ((n = sourceStream.Read(buffer, 0, buffer.Length)) != 0)
           targetStream.Write(buffer, 0, n);
}

Answer 2

The size of the buffer does matter. If you're copying a megabyte of data one byte at a time, for example, you're going to make a 2^20 iterations through the loop. If you're copying 1 kilobyte at a time, you'll only make 2^10 iterations through the loop. There is significant overhead in the calls to Read and Write when you're making a million of them.

For reading FileStream, I typically use a buffer that's between 64K and 256K. Anything less than 32K shows a marked decrease in performance, as does anything above 256K. The difference between using a 64K buffer and a 256K buffer is not worth the extra memory. Be aware, though, that those numbers are on my system and across my network. Your numbers will vary depending on hardware and operating system.

For network streams, you should select a buffer size that will keep up with the incoming data stream. I'd suggest at least 4 kilobytes, which will give you some buffer if the write stalls for any reason.