Force JVM to do all IO without page cache (e.g. O_DIRECT)

Question

I'm doing some benchmarks of an application written in Java. It is very important for the experiments that the results are not influenced by the page cache (I'm using linux)

So the best way to avoid the page cache is using O_DIRECT whenever a file is opened. I therefore changed the respective code in the sourcecode of the jre.

My approach works perfectly for everything that goes through the FileOutputStream (e.g. writing), but it does not work for FileInputStream (e.g. reading).

When adding O_DIRECT to the open-call of FileInputStream, the JVM is unable to load any classes:

Error: Could not find or load main class perf.TestDirectIO

This error is not a classpath issue, since I can fix it just by using a "unhacked" JVM.

So there seems to be an issue with opening files.

I'm very happy about any advice on how to fix the issue.

If anyone ever wants to do something similar, I've documented the whole hack in my blog.

---

As a reference, these are the changes on the JVM code I did so:

jdk/src/share/native/java/io/FileInputStream.c:

 @@ -58,7 +60,8 @@
 JNIEXPORT void JNICALL
 Java_java_io_FileInputStream_open(JNIEnv *env, jobject this, jstring path) {
-    fileOpen(env, this, path, fis_fd, O_RDONLY);
+    fileOpen(env, this, path, fis_fd, O_RDONLY | O_DIRECT); // this is the change that causes all the problems
 }

This change works: jdk/src/solaris/native/java/io/FileOutputStream_md.c:

@@ -55,8 +55,10 @@
 JNIEXPORT void JNICALL
 Java_java_io_FileOutputStream_open(JNIEnv *env, jobject this,
                                    jstring path, jboolean append) {
     fileOpen(env, this, path, fos_fd,
-             O_WRONLY | O_CREAT | (append ? O_APPEND : O_TRUNC));
+             O_WRONLY | O_DIRECT | O_CREAT | (append ? O_APPEND : O_TRUNC));
 }

I also changed the hotspot jre to ensure that the memory is aligned (thats a requirement for O_DIRECT) hotspot/src/share/vm/runtime/os.cpp:

+# include <mm_malloc.h>
...
-  u_char* ptr = (u_char*)::malloc(size + space_before + space_after);
+  u_char* ptr = (u_char*)::_mm_malloc(size + space_before + space_after,512);

Answer 1

Old post, but I recently wrote a small library called Jaydio hoping to solve this exact problem. Maybe you will find it useful.

Answer 2

  "The thing that has always disturbed me about O_DIRECT is that the
   whole interface is just stupid, and was probably designed by a deranged
   monkey on some serious mind-controlling substances  [*]."

[*] In other words, it's an Oracleism.

-- Linus Torvalds from Transmeta, 11 May 2002

Check NOTES section of the man 2 open:

O_DIRECT

The O_DIRECT flag may impose alignment restrictions on the length and address of userspace buffers and the file offset of I/Os. In Linux alignment restrictions vary by file system and kernel version ....

Under Linux 2.4, transfer sizes, and the alignment of the user buffer and the file offset must all be multiples of the logical block size of the file system. Under Linux 2.6, alignment to 512-byte boundaries suffices. ....

In summary, O_DIRECT is a potentially powerful tool that should be used with caution. It is recommended that applications treat use of O_DIRECT as a performance option which is disabled by default.

I think, there are some usages of FileInputStream in the JRE (classloader) which has reads with offsets or sizes not aligned to 512 bytes. (For Advanced Format the minimal alignment may be bigger, even 4096 bytes, or one 4K page.)

The behaviour of kernel for unaligned offsets is the grey zone, some info is here: RFC: Clarifying Direct I/O Semantics, Theodore Ts'o, tytso@mit, LWN, 2009

Other interesting discussion is here: Linux: Accessing Files With O_DIRECT (kerneltrap, 2007)

Hmm, seems like there should be fallback to buffered I/O when something with DIRECT fails. All IO operations with DIRECT are synchronous. May be some DMA effects? Or combination of O_DIRECT and mmap?

UPDATE:

Thanks for strace output. Here is the error (grep O_DIRECT, then check file descriptor operations):

28290 open("...pact/perf/TestDirectIO.class", O_RDONLY|O_DIRECT) = 11
28290 fstat(11, {st_mode=S_IFREG|0644, st_size=2340, ...}) = 0
28290 fcntl(11, F_GETFD)                = 0
28290 fcntl(11, F_SETFD, FD_CLOEXEC)    = 0
...skip
28290 stat("...pact/perf/TestDirectIO.class", {st_mode=S_IFREG|0644, st_size=2340, ...}) = 0
...skip
28290 read(11, "\312\376\272\276\0\0\0003\0\215\n\0-\0D\t\0E\0F\7\0G\n\0\3\0D\10\0H\n"..., 1024) = 1024
28290 read(11, 0x7f1d76d23a00, 1316)    = -1 EINVAL (Invalid argument)

Unaligned read size results in EINVAL error. Your classfile is 2340 bytes long, it is 1024+1316 bytes, which is not aligned.