O_DIRECT vs. O_SYNC on Linux/FreeBSD

Question

I'm writing a program that runs on both Linux and FreeBSD, and I want to make sure that the data is actually written to the file on the physical device when each write() returns, so that my data won't get lost by accident (eg, power lost, the process is interrupted unexpected, etc.).

According to OPEN(2) man page, on Linux (higher than 2.6), O_DIRECT is synchronous but may have performance problems; on FreeBSD, O_DIRECT is not guaranteed synchronous and may also has problems.

So, on Linux, either O_DIRECT or O_SYNC guarantees synchronous write, but which one has better performance?

On FreeBSD, to guarantee synchronous write, which option has the best performance: (1) O_DIRECT + fsync() (2) O_DIRECT | O_SYNC or (3)O_SYNC alone?

Answer 1

With current harddisks, there is no assurance that a file is actually written to disk even if the disk reports the write as complete to the OS! This is due to built-in cache in the drive.

On freeBSD you can disable this by setting the kern.cam.ada.write_cache sysctl to 0. This will degrade write performance significantly. Last time I measured it (WDC WD5001ABYS-01YNA0 harddisk on an ICH-7 chipset, FreeBSD 8.1 AMD64), continuous write performance (measured with dd if=/dev/zero of=/tmp/foo bs=10M count=1000) dropped from 75,000,000 bytes/sec to 12,900,000 bytes/sec.

If you want to be absolutely sure that your files are written;

• Disable write caching with sysctl kern.cam.ada.write_cache=0 followed by camcontrol reset <bus>:<target>:<lun>.
• Open the file with the O_SYNC option.

Note:

• Your write perfomance (on a HDD) will now absolutely suck.
• Do not mount the partition with the sync option; that will cause all I/O (including reads) to be done syncronously.
• Do not use O_DIRECT. It will try to bypass the cache altogether. That will probably also influence reads.

Answer 2

O_DIRECT basically exists solely for Oracle to bypass the kernel's buffer cache layer and do its own caching. It has ill-defined semantics, arbitrary limitations on the size and alignment of reads you can perform, and generally should not be used. O_SYNC is supposed to give you the effects you want, but without an underlying filesystem that's robust against power failure or crashes, it still might not be sufficient for your needs.