Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there an os-independent way to atomically overwrite a file?

Tags:

file

atomic

go

If the file already exists, I want to overwrite it. If it doesn't exist, I want to create it and write to it. I'd prefer to not have to use a 3rd party library like lockfile (which seems to handle all types of locking.)

My initial idea was to:

  1. Write to a temporary file with a randomly generated large id to avoid conflict.
  2. Rename the temp filename -> new path name.
like image 947
lf215 Avatar asked May 21 '15 22:05

lf215


People also ask

Is renaming a file Atomic?

In POSIX, a successful call to rename is guaranteed to have been atomic from the point of view of the current host (i.e., another program would only see the file with the old name or the file with the new name, not both or neither of them).

Is ReplaceFile Atomic?

aspx you'll see that ReplaceFile is a complicated merge operation, with no indication that it's atomic.

Is mv command Atomic?

mv is most definitely not atomic when the move that it performs is from one filesystem to another, or when a remote filesystem cannot implement the mv operation locally. In these instances mv could be said to be implemented by the equivalent of a cp followed by rm .


1 Answers

os.Rename calls syscall.Rename which for Linux/UNIXs uses the rename syscall (which is atomic*). On Windows syscall.Rename calls MoveFileW which assuming the source and destination are on the same device (which can be arranged) and the filesystem is NTFS (which is often the case) is atomic*.

I would take care to make sure the source and destination are on the same device so the Linux rename does not fail, and the Windows rename is actually atomic. As Dave C mentions above creating your temporary file (usually using ioutil.TempFile) in the same directory as existing file is the way to go; this is how I do my atomic renames.

This works for me in my use case which is:

  1. One Go process gets updates and renames files to swap updates in.
  2. Another Go process is watching for file updates with fsnotify and re-mmaps the file when it is updated.

In the above use case simply using os.Rename has worked perfectly well for me.

Some further reading:

  1. Is rename() atomic? "Yes and no. rename() is atomic assuming the OS does not crash...."
  2. Is an atomic file rename (with overwrite) possible on Windows?

*Note: I do want to point out that when people talk about atomic filesystem file operations, from an application perspective, they usually mean the operation happens or does not happen (which journaling can help with) from the users perspective. If you are using atomic in the sense of an atomic memory operation, very few filesystem operations (outside of direct I/O [O_DIRECT] one block writes and reads with disk buffering disabled) can be considered truly atomic.

like image 179
voidlogic Avatar answered Sep 30 '22 20:09

voidlogic