Search engine to integrate in a .NET stack

Question

I am looking for a very robust software search engine to integrate in a .Net web site.

The current proposed solution is Lucene.NET a stack based on Lucene. However, I would like to evaluate other search engines before making my mind up.

The feature set we need is the following:

• Ability to crawl arbitrary pages via HTTP
• Ability to parse sitemaps
• Ability to get lists of URIs to parse via a database look-up
• Ability to restrict the search to a particular language/locale
• Ability to restrict the search to a subset of the pages (e.g. via a regex on the URI)
• Speed and scalability (this is for a public website with a ton of traffic)
• Must have .NET API support or a super-easy http-based API that can be wrapped in a .NET API
• Language-dependent full-text support

Other things which would be great, but not deal-breakers if they aren't supported:

• Reporting
• Aliasing and biasing of results
• HTTP-based administration pages
• SQL Server support

What other software search engines have worked for you? Is there any you would recommend or that we should avoid?

Answer 1

Check out Microsoft's Search Server Express, although the page looks screwed up at the moment so try this link.

There's other enterprise engines out there such as vivisimo velocity (very extensible), autonomy, etc. Lucene and Solr are limited, hard to use and configure, but that's what you get when you want something free.

Answer 2

Lucene.Net is an information retrieval library, not a search engine. In particular it won't do any of:

• Crawl web pages or parse sitemaps
• Reporting
• HTTP-based administration pages
• SQL Server support (Lucene.Net uses its own simple but highly effective file format, and doesn't use SQL Server)

Although I'm a strong supporter or SQL and would recommend it as the full-text search component of a search engine, you will also need a crawler / html parser component in order to create a full functional search engine, and you are going to have to carefully design your Lucene.Net indexes to maximise the performance of the queries that you want (searching by language/locale)

Try looking at the Solr project which is a fully fledged search engine using Lucene - this might be more suited towards your needs.