Can I store Lucene Search Index‎ in Amazon elastic file system (EFS) as an alternative to a system directory

Question

I would like to know if it is possible to store Lucene Search Index‎ in Amazon elastic file system (EFS) as an alternative to a system directory provider(such as that used in hibernate search in java i.e hibernate.search.default.indexBase. And if so what is the best way to implement this. Thanks in advance?

Answer 1

So far there has been with issues on search and index performance degradation with Apache Lucene running on NFS. As EFS supports NFS. Storing Lucene search index on AWS NFS would most likely cause the Linux to lock up and give a load of error messages by default. By experience, In the long run EFS is based on NFS and it isn’t good(at least so far) to use it with lucene.

Answer 2

Have built a few large Hibernate / Lucene indexes - the primary issue is really file-locking. Performance is one thing wherein a lot of corporate NFS farms are general-purpose storage, not tuned as a backing store for a full-text search (implied "sub-second" or close thereof expected response time).

What happens pragmatically is that, for update-capable indexes as a use-case, is that to update an index, the file in question (and there are many files making up the indices) has to be exclusively locked system wide.

NFS has had a long history of locking issues (having programmed for them in "C" back in the day) with the NFS "lock daemon" and associated models wherein the process gets hung, stalls, has to be restarted, etc.