Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

File.Copy vs. Manual FileStream.Write For Copying File

My problem is in regards file copying performance. We have a media management system that requires a lot of moving files around on the file system to different locations including windows shares on the same network, FTP sites, AmazonS3, etc. When we were all on one windows network we could get away with using System.IO.File.Copy(source, destination) to copy a file. Since many times all we have is an input Stream (like a MemoryStream), we tried abstracting the Copy operation to take an input Stream and an output Stream but we are seeing a massive performance decrease. Below is some code for copying a file to use as a discussion point.

public void Copy(System.IO.Stream inStream, string outputFilePath) {     int bufferSize = 1024 * 64;      using (FileStream fileStream = new FileStream(outputFilePath, FileMode.OpenOrCreate, FileAccess.Write))     {          int bytesRead = -1;         byte[] bytes = new byte[bufferSize];          while ((bytesRead = inStream.Read(bytes, 0, bufferSize)) > 0)         {             fileStream.Write(bytes, 0, bytesRead);             fileStream.Flush();         }     } } 

Does anyone know why this performs so much slower than File.Copy? Is there anything I can do to improve performance? Am I just going to have to put special logic in to see if I'm copying from one windows location to another--in which case I would just use File.Copy and in the other cases I'll use the streams?

Please let me know what you think and whether you need additional information. I have tried different buffer sizes and it seems like a 64k buffer size is optimal for our "small" files and 256k+ is a better buffer size for our "large" files--but in either case it performs much worse than File.Copy(). Thanks in advance!

like image 462
jakejgordon Avatar asked Aug 07 '09 20:08

jakejgordon


People also ask

What is the use of FileStream?

Remarks. Use the FileStream class to read from, write to, open, and close files on a file system, and to manipulate other file-related operating system handles, including pipes, standard input, and standard output.

How do I retrieve a file from FileStream?

Read file using FileStreamFirst create FileStream to open a file for reading. Then call FileStream. Read in a loop until the whole file is read. Finally close the stream.


1 Answers

File.Copy was build around CopyFile Win32 function and this function takes lot of attention from MS crew (remember this Vista-related threads about slow copy performance).

Several clues to improve performance of your method:

  1. Like many said earlier remove Flush method from your cycle. You do not need it at all.
  2. Increasing buffer may help, but only on file-to-file operations, for network shares, or ftp servers this will slow down instead. 60 * 1024 is ideal for network shares, at least before vista. for ftp 32k will be enough in most cases.
  3. Help os by providing your caching strategy (in your case sequential reading and writing), use FileStream constructor override with FileOptions parameter (SequentalScan).
  4. You can speed up copying by using asynchronous pattern (especially useful for network-to-file cases), but do not use threads for this, instead use overlapped io (BeginRead, EndRead, BeginWrite, EndWrite in .net), and do not forget set Asynchronous option in FileStream constructor (see FileOptions)

Example of asynchronous copy pattern:

int Readed = 0; IAsyncResult ReadResult; IAsyncResult WriteResult;  ReadResult = sourceStream.BeginRead(ActiveBuffer, 0, ActiveBuffer.Length, null, null); do {     Readed = sourceStream.EndRead(ReadResult);      WriteResult = destStream.BeginWrite(ActiveBuffer, 0, Readed, null, null);     WriteBuffer = ActiveBuffer;      if (Readed > 0)     {       ReadResult = sourceStream.BeginRead(BackBuffer, 0, BackBuffer.Length, null, null);       BackBuffer = Interlocked.Exchange(ref ActiveBuffer, BackBuffer);     }      destStream.EndWrite(WriteResult);   }   while (Readed > 0); 
like image 81
arbiter Avatar answered Sep 16 '22 23:09

arbiter