How to design a customized Search Engine?

Question

I want to design my own search engine application, where all the results are displayed to the user on one single page (from Google/Bing etc) unlike Google where it is displayed on different pages.

Does there exist any such API's which can get me all those results?

PS. I am using C#, and considering the IEnumerator interface for this?

Answer 1

If you just want to be able to serve search results to users, then the APIs provided by search engines are probably the way to go. As already mentioned there's Bing's Live Search API (which I've not used but looks fine), and also Google's Web Search API. Additionally, there's Yahoo BOSS which I found very easy to use. However, it looks like BOSS is now a paid API - so depending on your budget/intention, it might not suit.

Google's Web Search API is now deprecated, but should still work for a small number of queries - it's the platform that tools like this number of results counter are built on. It's been replaced by the Google Custom Search API which depending on your needs may or may not work for you. I've not used it, but it looks fine, and is free for small numbers of queries.

The problem with crawling and then parsing search pages is that search engines regularly change the underlying html of the search result pages - so any screen scraping approach will be quite brittle. Additionally, the terms of service of most commercial search engines prohibit automated access - if you go ahead anyway they may well block your crawler. These two problems are probably why awesome third party parsing APIs don't really exist.