Paging Lucene's search results

Question

I am using Lucene to show search results in a web application.I am also custom paging for showing the same. Search results could vary from 5000 to 10000 or more. Can someone please tell me the best strategy for paging and caching the search results?

Answer 1

I would recommend you don't cache the results, at least not at the application level. Running Lucene on a box with lots of memory that the operating system can use for its file cache will help though.

Just repeat the search with a different offset for each page. Caching introduces statefulness that, in the end, undermines performance. We have hundreds of concurrent users searching an index of over 40 million documents. Searches complete in much less than one second without using explicit caching.

Using the Hits object returned from search, you can access the documents for a page like this:

Hits hits = searcher.search(query);
int offset = page * recordsPerPage;
int count = Math.min(hits.length() - offset, recordsPerPage);
for (int i = 0; i < count; ++i) {
  Document doc = hits.doc(offset + i);
  ...
}