I'm looking for a stand-alone full-text search server with the following properties:
- Must operate as a stand-alone server that can serve search requests from multiple clients
- Must be able to do "bulk indexing" by indexing the result of an SQL query: say "SELECT id, text_to_index FROM documents;"
- Must be free software and must run on Linux with MySQL as the database
- Must be fast (rules out MySQL's internal full-text search)
The alternatives I've found that have these properties are:
- Solr (based on Lucene)
- ElasticSearch (also based on Lucene)
- Sphinx
My questions:
- How do they compare?
- Have I missed any alternatives?
- I know that each use case is different, but are there certain cases where I would definitely not want to use a certain package?
Should I use Solr or Lucene?
A simple way to conceptualize the relationship between Solr and Lucene is that of a car and its engine. You can't drive an engine, but you can drive a car. Similarly, Lucene is a programmatic library which you can't use as-is, whereas Solr is a complete application which you can use out-of-box.
What is the difference between Solr and Elasticsearch?
Solr had a broad, open source community. Anyone can still contribute to Solr, and new Solr developers or code committers are elected based on merit only. Elasticsearch is technically open source but not fully. All contributors have access to the source code, and users can make changes and contribute them.
Can Solr be used as a database?
Solr is a search engine at heart, but it is much more than that. It is a NoSQL database with transactional support. It is a document database that offers SQL support and executes it in a distributed manner.
What is Solr searching?
Apache Solr (stands for Searching On Lucene w/ Replication) is a free, open-source search engine based on the Apache Lucene library. An Apache Lucene subproject, it has been available since 2004 and is one of the most popular search engines available today worldwide.
Unless you need to extend the search functionality in any proprietary way, Sphinx is your best bet.
Sphinx advantages:
- Development and setup is faster
- Much better (and faster) aggregation. This was the killer feature for us.
- Not XML. This is what ultimately ruled out Solr for us. We had to return rather large result sets (think hundreds of results) and then aggregate them ourselves since Solr aggregation was lacking. The amount of time to serialize to and from XML just absolutely killed performance. For small results sets though, it was perfectly fine.
- Best documentation I've seen in an open source app
Solr advantages:
- Can be extended.
- Can hit it directly from a web app, i.e., you can have autocomplete-like searches hit the Solr server directly via AJAX.
I've been using Solr successfully for almost 2 years now, and have never used Sphinx, so I'm obviously biased. However, I'll try to keep it objective by quoting the docs or other people. I'll also take patches to my answer :-)
Similarities:
- Both Solr and Sphinx satisfy all of your requirements. They're fast and designed to index and search large bodies of data efficiently.
- Both have a long list of high-traffic sites using them (Solr, Sphinx)
- Both offer commercial support. (Solr, Sphinx)
- Both offer client API bindings for several platforms/languages (Sphinx, Solr)
- Both can be distributed to increase speed and capacity (Sphinx, Solr)
Here are some differences:
- Solr, being an Apache project, is obviously Apache2-licensed. Sphinx is GPLv2. This means that if you ever need to embed or extend (not just "use") Sphinx in a commercial application, you'll have to buy a commercial license (rationale)
- Solr is easily embeddable in Java applications.
- Solr is built on top of Lucene, which is a proven technology over 8 years old with a huge user base (this is only a small part). Whenever Lucene gets a new feature or speedup, Solr gets it too. Many of the devs committing to Solr are also Lucene committers.
- Sphinx integrates more tightly with RDBMSs, especially MySQL.
- Solr can be integrated with Hadoop to build distributed applications
- Solr can be integrated with Nutch to quickly build a fully-fledged web search engine with crawler.
- Solr can index proprietary formats like Microsoft Word, PDF, etc. Sphinx can't.
- Solr comes with a spell-checker out of the box.
- Solr comes with facet support out of the box. Faceting in Sphinx takes more work.
- Sphinx doesn't allow partial index updates for field data.
- In Sphinx, all document ids must be unique unsigned non-zero integer numbers. Solr doesn't even require an unique key for many operations, and unique keys can be either integers or strings.
- Solr supports field collapsing (currently as an additional patch only) to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.
- While Sphinx is designed to only retrieve document ids, in Solr you can directly get whole documents with pretty much any kind of data, making it more independent of any external data store and it saves the extra roundtrip.
- Solr, except when used embedded, runs in a Java web container such as Tomcat or Jetty, which require additional specific configuration and tuning (or you can use the included Jetty and just launch it with
java -jar start.jar). Sphinx has no additional configuration.
Related questions:
- Full Text Searching with Rails
- Comparison of full text search engine - Lucene, Sphinx, Postgresql, MySQL?
Menu
687
