Atomic write to multiple files

Question

Suppose I have a set of files. How do I ensure that writing to all these files is atomic.

I thought about writing to temp files and only after the writing is successful, perform an atomic rename of each file. However renaming all the files at once isn't atomic. Also this will not scale to very large files if we'd like to append to these files.

Instead I thought about implementing transactions but then that's becoming a project on its own. I realize that this is pretty much about implementing a mini database.

How would you do it in Python?

d = FileWriter.open(['file1', 'file2'], 'wb+')
d.write('add hello world to files')
d.close()

Ensure that d.write is atomic or at least rollback to original files if unsuccessful.

Answer 1

Here is what I'm thinking. First ensure open is synchronized, then perform the following:

1. Write to temp files: file1~, file2~ and special file success~ (must be written first).
2. After a successful write, remove file success~.
3. Rename the files to file1 and file2.

If something breaks:

1. Check if success~ exists.
2. If it does, do not bother repairing. A rollback was implicitly performed because the files were not updated (no renames).
3. If success~ does not exist, things broke after writing and during renaming. In this case repairing is as simple as renaming filex~ to filex.

Answer 2

Your requirement, in its current statement, is so generic that the only plausible answer is that OS level support is necessary, and it's not even strictly a Python-related question anymore. See what is said about Transactional file systems, for example here and here. A small excerpt regarding the solutions proposed so far:

Ensuring consistency across multiple file system operations is difficult, if not impossible, without file system transactions. File locking can be used as a concurrency control mechanism for individual files, but it typically does not protect the directory structure or file metadata. For instance, file locking cannot prevent TOCTTOU race conditions on symbolic links. File locking also cannot automatically roll back a failed operation, such as a software upgrade; this requires atomicity.

So I would suggest to rethink your problem (and maybe your question too?) and analyze your requirements with greater detail. The bottom line is that the less guarantees you need, the simpler solution you can find. Maybe you can get away with something as simple as file locking, or maybe you will find that a database will be needed.

Speaking of which, if you're thinking about a filesystem because the components of your architecture need to access files, have you thought about FUSE as a way to build a filesystem-like facade on top of a regular database? Using python-fuse is a breeze.