Force write of a file to disk

Question

I'm currently implementing a ping/pong buffering scheme to safely write a file to disk. I'm using C++/Boost on a Linux/CentOS machine. Now I'm facing the problem to force the actual write of the file to disk. Is it possible to do so irrespective of all the caching policies of the filesystem (ext3/ext4) / SO custom rules / RAID controller / harddisk controller ?

Is it best to use plain fread()/fwrite(), c++ ostream or boost filesystem?

I've heard that simply flushing out the file (fflush()) doesn't guarantee the actual write

Answer 1

fflush (for FILE*), std::flush (for IOStream) to force your program to send to the OS.

POSIX has

• sync(2) to ask to schedule writing its buffers, but can return before the writing is done (Linux is waiting that the data is send to the hardware before returning).

• fsync(2) which is guaranteed to wait for the data to be send to the hardware, but needs a file descriptor (you can get one from a FILE* with fileno(3), I know of no standard way to get one from an IOStream).

• O_SYNC as a flag to open(2).

In all cases, the hardware may have it's own buffers (but if it has control on it, a good implementation will try to flush them also and ISTR that some disks are using capacitors so that they are able to flush whatever happens to the power) and network file systems have their own caveat.

Answer 2

You can use fsync()/fdatasync() to force(Note 1) the data onto the storage. Those requres a file descriptor, as given by e.g. open(). The linux manpage have more linux specific info, particularly on the difference of fsync and fdatasync.

If you don't use file desciptors directly, many abstractions will contain internal buffers residing in your process.

e.g. if you use a FILE*, you first have to flush the data out of your application.

//... open and write data to a FILE *myfile
fflush(myfile);
fsync(fileno(myfile));

• Note 1: These calls force the OS to ensure that any data in any OS cache is written to the drive, and the drive acknowledges that fact. Many hard-drives lie to the OS about this, and might stuff the data in cache memory on the drive.