Reliable file saving (File.Replace) in a busy environment

Question

I am working on server software that periodically needs to save data to disk. I need to make sure that the old file is overwritten, and that the file cannot get corrupted (e.g. only partially overwritten) in case of unexpected circumstances.

I've adopted the following pattern:

string tempFileName = Path.GetTempFileName(); // ...write out the data to temporary file... MoveOrReplaceFile(tempFileName, fileName); 

...where MoveOrReplaceFile is:

public static void MoveOrReplaceFile( string source, string destination ) {     if (source == null) throw new ArgumentNullException("source");     if (destination == null) throw new ArgumentNullException("destination");     if (File.Exists(destination)) {         // File.Replace does not work across volumes         if (Path.GetPathRoot(Path.GetFullPath(source)) == Path.GetPathRoot(Path.GetFullPath(destination))) {             File.Replace(source, destination, null, true);         } else {             File.Copy(source, destination, true);         }     } else {         File.Move(source, destination);     } } 

This works well as long as the server has exclusive access to files. However, File.Replace appears to be very sensitive to external access to files. Any time my software runs on a system with an antivirus or a real-time backup system, random File.Replace errors start popping up:

System.IO.IOException: Unable to remove the file to be replaced.

Here are some possible causes that I've eliminated:

• Unreleased file handles: using() ensures that all file handles are released as soon as possible.
• Threading issues: lock() guards all access to each file.
• Different disk volumes: File.Replace() fails when used across disk volumes. My method checks this already, and falls back to File.Copy().

And here are some suggestions that I've come across, and why I'd rather not use them:

• Volume Shadow Copy Service: This only works as long as the problematic third-party software (backup and antivirus monitors, etc) also use VSS. Using VSS requires tons of P/Invoke, and has platform-specific issues.
• Locking files: In C#, locking a file requires maintaining a FileStream open. It would keep third-party software out, but 1) I still won't be able to replace the file using File.Replace, and 2) Like I mentioned above, I'd rather write to a temporary file first, to avoid accidental corruption.

I'd appreciate any input on either getting File.Replace to work every time or, more generally, saving/overwriting files on disk reliably.

Answer 1

You really want to use the 3rd parameter, the backup file name. That allows Windows to simply rename the original file without having to delete it. Deleting will fail if any other process has the file opened without delete sharing, renaming is never a problem. You could then delete it yourself after the Replace() call and ignore an error. Also delete it before the Replace() call so the rename won't fail and you'll cleanup failed earlier attempts. So roughly:

string backup = destination + ".bak"; File.Delete(backup); File.Replace(source, destination, backup, true); try {     File.Delete(backup); } catch {     // optional:     filesToDeleteLater.Add(backup); } 

Answer 2

There are several possible approaches, here some of them:

1. Use a "lock" file - a temporary file that is created before the operation and indicates other writers (or readers) that the file is being modified and thus exclusively locked. After the operation complete - remove the lock file. This method assumes that the file-creation command is atomic.
2. Use NTFS transactional API (if appropriate).
3. Create a link to the file, write the changed file under a random name (for example Guid.NewGuid()) - and then remap the link to the new file. All readers will access the file through the link (which name is known).

Of course all 3 approaches have their own drawbacks and advantages