How to load an FSdirectory index file as a RAMdirectory in Lucene?

Question

I have an index file in hard as FSdirectory which size is about 2GB. I want to load it in ram to have a better search speed. I use the code behind:

Directory dic=new RAMDirectory(FSDirectory.open(new File("file path")), IOContext.DEFAULT);

But I'm not sure if it work. Do anyone know that it work or not? Any other way better than this on?

I'll be glad of any suggestion.Thanks.

Answer 1

Yes you can construct RAMDirectory like that but it's strongly discouraged in your case. RAMDirectory is not designed to be used with large data set (more than 100MB). It will be very GC intensive, long to construct and consume precious JAVA Heap space.

If you have perf issues please make sure that you follow all the guidelines descibed in this lucene FAQ

Don't forget that to get descent search speed the os needs to cache some part of the index into memory. In order to do that you must let some free mem available to the OS. If you have 16Gb of physical RAM and a 8gb index you should configure the jvm Xmx with 8g. A nice linux tool to check mem is htop, it will print RAM used for process as green (your heap space) and RAM used as filecache as yellow (the index data).

Some developpers perform warm up queries at application startup to prevent the first user from being stuck few seconds while the index gets loaded by os filecache.

Then if you still think you can get better perf you can try some tricks with RAMfs. But I doubt that you can get better performances than a well tuned os and jvm with MMapDirectory.