Menu
I need a way to store large data chunks(~1-2MB) at a high rate (~200-300Mbit/s).
After some research I found several options:
aio_writePBWriteForkAsync()
fwrite(), wrapped in a block and dispatched via GCDappendData in an NSOperationThis wiki page describes the state of aio_write under Linux. What I didn't find was a similar page about the state of aio_write for Mac OS X.
NSOperation or Blocks+GCD seems to be a technique to achieve non-blocking IO. It is used in several open source IO libraries (e.g. https://github.com/mikeash/MAAsyncIO)
Has someone with a similar problem found a suitable solution?
Currently I tend towards PBWriteForkAsync as it takes some 'tuning'parameters. It also should be 64-bit safe.
585
Drives that are used primarily for large multimedia files or backups should have relatively high sequential write speeds. When manufacturers quote speeds for devices such as flash drives, hard drives and SSD's, unless otherwise specified, they are referring to the sequential speed.
The file is opened... Data is 'streamed' to disk. The data in memory is in a large contiguous buffer. It is written to disk in its raw form directly from that buffer. The size of the buffer is configurable, but fixed for the duration of the stream. Buffers are written to the file, one after another.
Are there generally accepted guidelines for achieving the fastest possible sequential file I/O in C++? Rule 0: Measure. Use all available profiling tools and get to know them. It's almost a commandment in programming that if you didn't measure it you don't know how fast it is, and for I/O this is even more true.
BinaryReader and BinaryWriter with a suitable buffer size are pretty fast. If you are reading into structures, the unsafe approach described in this article will get you reading fast, and writing is similar. I also agree with the suggestion to double-check that I/O is really the bottleneck.
I don't know MacOS very well, but I'd also try open and write syscalls from unistd.h with the non-blocking option O_NONBLOCK. reference
67
You should use unbuffered I/O for the writes, in Carbon this is FSWriteFork() with kFSNoCacheBit, in BSD use fcntl() with F_NOCACHE.
Rather than use the system's non-blocking IO, you may want to consider a worker thread to write the blocks sequentially using a queue. This will give you more control and may end up being simpler, particularly if you want to monitor the queue to see if you are keeping up or not.
See here for more information.
24
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
