An alternative web crawler to Nutch [closed]

Question

I'm trying to build a specialised search engine web site that indexes a limited number of web sites. The solution I came up with is:

• using Nutch as the web crawler,
• using Solr as the search engine,
• the front-end and the site logic is coded with Wicket.

The problem is that I find Nutch quite complex and it's a big piece of software to customise, despite the fact that a detailed documentation (books, recent tutorials.. etc) does just not exist.

Questions now:

1. Any constructive criticism about the hole idea of the site?
2. Is there a good yet simple alternative to Nutch (as the crawling part of the site)?

Thanks

Answer 1

Scrapy is a python library that crawls web sites. It is fairly small (compared to Nutch) and designed for limited site crawls. It has a Django type MVC style that I found pretty easy to customize.

Answer 2

For the crawling part, I really like anemone and crawler4j. They both allow you to add your custom logic for links selection and page handling. For each page that you decide to keep, you can easily add the call to Solr.