How can I copy a large file on Windows without CopyFile or CopyFileEx?

Question

There is a limitation on Windows Server 2003 that prevents you from copying extremely large files, in proportion to the amount of RAM you have. The limitation is in the CopyFile and CopyFileEx functions, which are used by xcopy, Explorer, Robocopy, and the .NET FileInfo class.

Here is the error that you get:

Cannot copy [filename]: Insufficient system resources exist to complete the requested service.

The is a knowledge base article on the subject, but it pertains to NT4 and 2000.

There is also a suggestion to use ESEUTIL from an Exchange installation, but I haven't had any luck getting that to work.

Does anybody know of a quick, easy way to handle this? I'm talking about >50Gb on a machine with 2Gb of RAM. I plan to fire up Visual Studio and just write something to do it for me, but it would be nice to have something that was already out there, stable and well-tested.

[Edit] I provided working C# code to accompany the accepted answer.

Answer 1

The best option is to just open the original file for reading, the destination file for writing and then loop copying it block by block. In pseudocode :

f1 = open(filename1);
f2 = open(filename2, "w");
while( !f1.eof() ) {
  buffer = f1.read(buffersize);
  err = f2.write(buffer, buffersize);
  if err != NO_ERROR_CODE
    break;
}
f1.close(); f2.close();

[Edit by Asker] Ok, this is how it looks in C# (it's slow but it seems to work Ok, and it gives progress):

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

namespace LoopCopy
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length != 2)
            {
                Console.WriteLine(
                  "Usage: LoopCopy.exe SourceFile DestFile");
                return;
            }

            string srcName = args[0];
            string destName = args[1];

            FileInfo sourceFile = new FileInfo(srcName);
            if (!sourceFile.Exists)
            {
                Console.WriteLine("Source file {0} does not exist", 
                    srcName);
                return;
            }
            long fileLen = sourceFile.Length;

            FileInfo destFile = new FileInfo(destName);
            if (destFile.Exists)
            {
                Console.WriteLine("Destination file {0} already exists", 
                    destName);
                return;
            }

            int buflen = 1024;
            byte[] buf = new byte[buflen];
            long totalBytesRead = 0;
            double pctDone = 0;
            string msg = "";
            int numReads = 0;
            Console.Write("Progress: ");
            using (FileStream sourceStream = 
              new FileStream(srcName, FileMode.Open))
            {
                using (FileStream destStream = 
                    new FileStream(destName, FileMode.CreateNew))
                {
                    while (true)
                    {
                        numReads++;
                        int bytesRead = sourceStream.Read(buf, 0, buflen);
                        if (bytesRead == 0) break; 
                        destStream.Write(buf, 0, bytesRead);

                        totalBytesRead += bytesRead;
                        if (numReads % 10 == 0)
                        {
                            for (int i = 0; i < msg.Length; i++)
                            {
                                Console.Write("\b \b");
                            }
                            pctDone = (double)
                                ((double)totalBytesRead / (double)fileLen);
                            msg = string.Format("{0}%", 
                                     (int)(pctDone * 100));
                            Console.Write(msg);
                        }

                        if (bytesRead < buflen) break;

                    }
                }
            }

            for (int i = 0; i < msg.Length; i++)
            {
                Console.Write("\b \b");
            }
            Console.WriteLine("100%");
            Console.WriteLine("Done");
        }
    }
}

Answer 2

If you want to write code, one way you can optimize is sending the file in chunks (like using MTOM). I used this approach for sending down huge files from a DataCenter down to our office for printing..

Also, check the TeraCopy utility mentioned here..