Reading Stream to a MemoryStream in multiple threads

Question

I am stuck up in a place. I am reading a flv file from a URL. I am reading this to a Stream and then writing this Stream to a MemoryStream in a loop. When the code comes out of the loop, I am writing the whole MemoryStream to a ByteArray and then writing this ByteArray to a local file on my hard disk.

As this flv is too large, it takes a lot of time to process in the loop. I am thinking of reading the original large stream in MemoryStream in multiple threads. That means dividing the Stream in say 10 parts and writing these parts to MemoryStream in multiple threads. How do I do this?

I am attaching my piece of code.

//Get a data stream from the url
                WebRequest req = WebRequest.Create(url);
                WebResponse response = req.GetResponse();
                using (Stream stream = response.GetResponseStream())
                {
                    //Download in chuncks
                    byte[] buffer = new byte[1024];

                    //Get Total Size
                    int dataLength = (int)response.ContentLength;



                    //Download to memory
                    //Note: adjust the streams here to download directly to the hard drive
                    using (MemoryStream memStream = new MemoryStream())
                    {
                        while (true)
                        {
                            //Try to read the data
                            int bytesRead = stream.Read(buffer, 0, buffer.Length);

                            if (bytesRead == 0)
                            {
                                Application.DoEvents();
                                break;
                            }
                            else
                            {
                                //Write the downloaded data
                                memStream.Write(buffer, 0, bytesRead);
                            }
                        }

                        //Convert the downloaded stream to a byte array
                        byte[] downloadedData = memStream.ToArray();
                    }  


                }

Any help is appreciated Thanks

Answer 1

You won't be able to speed up the download by using multiple threads. The limiting factor here is not how fast your computer can process the data, but rather how fast the data comes from the server.

Rather than try to speed this up using multiple threads, I would suggest that you create a WebClient rather than WebRequest. You can then call WebClient.DownloadDataAsync to download data into memory in the background, or call WebClient.DownloadFileAsync to download directly to a file.

Neither one of those will make the download any faster, but they will prevent your user interface from being non-responsive during the download.

Answer 2

Threads will not help you here; you are going to be blocked on IO. Rather than 1 thread blocked on IO, you will now have multiple threads blocked on IO. In fact, in many cases talking to the same resource (or parallel but related resources) on multiple threads will decrease IO throughput, plus the threading overheads. Lose : lose.

Also - most streams are not designed for threading; you would need some very complex co-ordination code to make sure you reassemble the stream in the right order and don't mess up the internal state; frankly, it isn't worth it.