File.Copy error: The requested operation could not be completed due to a file system limitation

Question

After getting more than 4000000 jpeg files in structure we've problem with adding new. File.Copy throw an exception: The requested operation could not be completed due to a file system limitation.

Any solution?

Info

• System: Windows Server 2008 SP1 x64
• Installed path: http://support.microsoft.com/kb/967351/en-us?fr=1
• Defragmentation: done

Code

    public bool AddFile(Uri uri, string path, bool withDelete = false)
    {
        var sourceFilePath = path;
        var destinationFilePath = Path.GetFullPath(uri.LocalPath);

        try
        {
            if (!File.Exists(sourceFilePath))
            {
                sourceFilePath = Directory.EnumerateFiles(sourceFilePath).FirstOrDefault();
                destinationFilePath = Path.Combine(destinationFilePath, Path.GetFileName(sourceFilePath));
            }

            if (!Directory.Exists(Path.GetDirectoryName(destinationFilePath)))
                Directory.CreateDirectory(Path.GetDirectoryName(destinationFilePath));

            if (withDelete && File.Exists(destinationFilePath))
                File.Delete(destinationFilePath);

            File.Copy(sourceFilePath, destinationFilePath);

            return true;
        }
        catch (Exception exc)
        {
            ServiceCore.GetLogger().Error(exc);
            throw exc;
        }
    }

Stacktrace

    2013-03-28 14:10:48.3784[Info]: 47356388:Unive.NetService.SimpleServices.DocumentManagementSerivce..ctor: Entry
    2013-03-28 14:10:48.4740[Info]: Static:Unive.NetService.SimpleServices.DocumentManagementSerivce..ctor: Success
    2013-03-28 14:10:48.4899[Info]: 47356388:Unive.NetService.SimpleServices.DocumentManagementSerivce.UploadFile: Entry
    2013-03-28 14:11:26.3277[Error]: Exception
    Message:The requested operation could not be completed due to a file system limitation

    Source:mscorlib
    Stack Trace:   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
       at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite)
       at Unive.NetService.Business.SimpleFileClient.AddFile(Uri uri, String path, Boolean withDelete) in D:\Tag Prografix\Unive.NetService\Business\SimpleFileClient.cs:line 33
    TargetSite:Void WinIOError(Int32, System.String)
    2013-03-28 14:11:26.5029[Error]:                         47356388:Unive.NetService.SimpleServices.DocumentManagementSerivce.UploadFileException
    Message:The requested operation could not be completed due to a file system limitation

    Source:mscorlib
    Stack Trace:   at Unive.NetService.Business.SimpleFileClient.AddFile(Uri uri, String path, Boolean withDelete) in D:\Tag Prografix\Unive.NetService\Business\SimpleFileClient.cs:line 42
       at Unive.NetService.Business.FileService.UploadFile(Int64 fileId, String fileName, String path, Boolean isDiagram) in D:\Tag Prografix\Unive.NetService\Business\FileService.cs:line 80
       at Unive.NetService.SimpleServices.DocumentManagementSerivce.UploadFile(Int64 fileId, String fileName, String path) in D:\Tag Prografix\Unive.NetService\SimpleServices\DocumentManagementSerivce.asmx.cs:line 100
    TargetSite:Void WinIOError(Int32, System.String)

Answer 1

You should divide and split your directory to avoid problems. Windows doesn't like directory with millions of files.

To avoid this problem, my files are always named with the db row ID (which is a guid).
And then each part of the guid is a directory, except the last one: The file with ID 02510b5a-a605-4a4e-b00a-f554998378a9 is stored in directory 02510b5a/a605/4a4e/b00a/ and have the name f554998378a9. So I can reach the file directly just using the ID, and millions of files are splitted in a lot of directories.

EDIT: Just a remark I notice about my solution since I posted here: In .NET Guid are generated so that the first part changes very often, and the last part not so often (or rarely). Using the split as described above lead to a lot of 1st level directory, then only 1 subdirectory in each subdirectories etc. So this still create a lot of 1st level directories and you could also reach the system limit (I don't know where is the limit, but Windows will surely not like to have 4 000 000 subdirectories in the same directory)

Solution: The solution is to revert to Guid parts when creating the directory.

Example: for this Guid 02510b5a-a605-4a4e-b00a-f554998378a9 you should use the directory f554998378a9\b00a\4a4e\a605 and filename 02510b5a

Be aware that .NET Guid are generated using the current time, so if you create millions of Guid in a loop they will all looks like the same (just the 1st part will be different) and ends up in the same directory using my solution.