Automatically selecting buffer size for File I/O

Question

I have a pretty basic doubt. Often, I have to write apps which use buffered file I/O and every time I am faced with the dilemma of choosing the buffer size and I end up doing trial and error often with pretty nasty results. I want to know if there is any method or algorithm which can automatically determine the optimum buffer size for the job based on the underlying platform like Teracopy does when handling files in Windows. I mainly use Qt for the GUI.

If possible a tiny example in C/C++/C#/Java is very much appreciated!

Thanks!

Answer 1

In Java the optimal is usually around the L1 cache size which is typically 32 KB. In Java, at least choosing 1024 bytes or 1 MB doesn't make much difference (<20%)

If you are reading data sequentially, usually your OS is smart enough to detect this and prefetch the data for you.

What you can do is the following. This test appears to show a significant difference in the block sizes used.

public static void main(String... args) throws IOException {
    for (int i = 512; i <= 2 * 1024 * 1024; i *= 2)
        readWrite(i);
}

private static void readWrite(int blockSize) throws IOException {
    ByteBuffer bb = ByteBuffer.allocateDirect(blockSize);
    long start = System.nanoTime();
    FileChannel out = new FileOutputStream("deleteme.dat").getChannel();
    for (int i = 0; i < (1024 << 20); i += blockSize) {
        bb.clear();
        while (bb.remaining() > 0)
            if (out.write(bb) < 1) throw new AssertionError();
    }
    out.close();
    long mid = System.nanoTime();
    FileChannel in = new FileInputStream("deleteme.dat").getChannel();
    for (int i = 0; i < (1024 << 20); i += blockSize) {
        bb.clear();
        while (bb.remaining() > 0)
            if (in.read(bb) < 1) throw new AssertionError();
    }
    in.close();
    long end = System.nanoTime();
    System.out.printf("With %.1f KB block size write speed %.1f MB/s, read speed %.1f MB/s%n",
            blockSize / 1024.0, 1024 * 1e9 / (mid - start), 1024 * 1e9 / (end - mid));
}

prints

With 0.5 KB block size write speed 96.6 MB/s, read speed 169.7 MB/s
With 1.0 KB block size write speed 154.2 MB/s, read speed 312.2 MB/s
With 2.0 KB block size write speed 201.5 MB/s, read speed 438.7 MB/s
With 4.0 KB block size write speed 288.0 MB/s, read speed 733.9 MB/s
With 8.0 KB block size write speed 318.4 MB/s, read speed 711.8 MB/s
With 16.0 KB block size write speed 540.6 MB/s, read speed 1263.7 MB/s
With 32.0 KB block size write speed 726.0 MB/s, read speed 1370.9 MB/s
With 64.0 KB block size write speed 801.8 MB/s, read speed 1536.5 MB/s
With 128.0 KB block size write speed 857.5 MB/s, read speed 1539.6 MB/s
With 256.0 KB block size write speed 794.0 MB/s, read speed 1781.0 MB/s
With 512.0 KB block size write speed 676.2 MB/s, read speed 1221.4 MB/s
With 1024.0 KB block size write speed 886.3 MB/s, read speed 1501.5 MB/s
With 2048.0 KB block size write speed 784.7 MB/s, read speed 1544.9 MB/s

What this test doesn't show is that the hard drive only supports 60 MB/s reads and 40 MB/s writes. All you are testing is the speed in and out of cache. If this was your only priority, you would use a memory mapped file.

int blockSize = 32 * 1024;
ByteBuffer bb = ByteBuffer.allocateDirect(blockSize);
FileChannel out = new FileOutputStream("deleteme.dat").getChannel();
for (int i = 0; i < (1024 << 20); i += blockSize) {
    bb.clear();
    while (bb.remaining() > 0)
        if (out.write(bb) < 1) throw new AssertionError();
}
out.close();

long start = System.nanoTime();
FileChannel in = new FileInputStream("deleteme.dat").getChannel();
MappedByteBuffer map = in.map(FileChannel.MapMode.READ_ONLY, 0, in.size());
in.close();
long end = System.nanoTime();
System.out.printf("Mapped file at a rate of %.1f MB/s%n",
        1024 * 1e9 / (end - start));

prints

Mapped file at a rate of 589885.5 MB/s

This is so fast because it just maps the data in the OS disk cache directly into the memory of the application (so no copying is required)

Answer 2

I have see this code in C:

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>

int main()
{
  struct stat fi;
  stat("/", &fi);
  printf("%d\n", fi.st_blksize);
  return 0;
}

It return the optimal block size. You need use it to do that's. I use stream source to destination with 16*block size to have optimal performance. Because this test will reveal the best with a pc at idle with some hardware/OS. But not real case.