Structured and Unstructured indexing - Lucene and Hbase

Question

I have a set of 200M documents I need to index. Every document has a free text and additional set of sparse metadata information (100+ columns).

It seems that the right tool for free text indexing is Lucene while the right tool for structured sparse metadata is HBase.

I would need to query the data and join between free text search results and the structured data results (e.g. get all books that has the phrase "good morning" in their textand were first published in 1980).

What tools/mechanism should I look at to join structured and unstrcutured queries? Results may include millions of records (before and after the join)

Thanks Saar

Answer 1

A couple of things come to mind, in addition to lucene on hbase:

1) Solr/Lucene can store multiple fields, and each field can have different types. So your date range example is plausible wholly within Solr.

2) If you are talking about truly huge data sets that require a cluster, also look at ElasticSearch: http://www.elasticsearch.org/

3) Lily attempts to answer your exact question http://www.lilyproject.org/lily/index.html

Answer 2

Looks like HBase would like some Lucene action as well: https://issues.apache.org/jira/browse/HBASE-3529.