Using ElasticSearch and/or Solr as a datastore for MS Office and PDF documents

Question

I'm currently designing a full text search system where users perform text queries against MS Office and PDF documents, and the result will return a list of documents that best match the query. The user will then be to select any document returned and view that document within MS Word, Excel, or a PDF viewer.

Can I use ElasticSearch or Solr to import the raw binary documents (ie. .docx, .xlsx, .pdf files) into its "data store", and then export the document to the user's device on command for viewing.

Previously, I used MongoDB 2.6.6 to import the raw files into GridFS and the extracted text into a separate collection (the collection contained a text index) and that worked fine. However, MongoDB full text searching is quite basic and therefore I'm now looking at either Solr or ElasticSearch to perform more complex text searching.

Nick

Answer 1

Both Solr and Elasticsearch will index the content of the document. Solr has that built-in, Elasticsearch needs a plugin. Easy either way and both use Tika under the covers.

Neither of them will store the document itself. You can try making them do it, but they are not designed for it and you will suffer.

Additionally, neither Solr nor Elasticsearch are currently recommended as a primary storage. They can do it, but it is not as mission critical for them as - say - for a filesystem implementation.

So, I would recommend having the files somewhere else and using Solr/Elasticsearch for searching only. That's where they shine.

Answer 2

I would try the Elasticsearch attachment plugin. Details can be found here:

https://www.elastic.co/guide/en/elasticsearch/plugins/2.2/mapper-attachments.html

https://github.com/elasticsearch/elasticsearch-mapper-attachments

It's built on top of Apache Tika:

http://tika.apache.org/1.7/formats.html

Attachment Type

The attachment type allows to index different "attachment" type field (encoded as base64), for example, Microsoft Office formats, open document formats, ePub, HTML, and so on (full list can be found here).

The attachment type is provided as a plugin extension. The plugin is a simple zip file that can be downloaded and placed under $ES_HOME/plugins location. It will be automatically detected and the attachment type will be added.

Supported Document Formats

• HyperText Markup Language

• XML and derived formats

• Microsoft Office document formats
• OpenDocument Format
• iWorks document formats
• Portable Document Format
• Electronic Publication Format
• Rich Text Format
• Compression and packaging formats
• Text formats
• Feed and Syndication formats
• Help formats
• Audio formats
• Image formats
• Video formats
• Java class files and archives
• Source code
• Mail formats
• CAD formats
• Font formats
• Scientific formats
• Executable programs and libraries
• Crypto formats