Menu
What is the best Open Source Web Crawler Tool, written in Java.
876
Apache Nutch is unquestionably at the top of the web crawler tool heap when it comes to the greatest open source web crawlers. Apache Nutch is a prominent open source code web data extraction software project for data mining that is highly flexible and scalable.
What are open source crawlers? Web crawlers are a type of software that automatically targets online websites and pulls their data in a machine-readable format. Open source web crawlers enable users to: modify the code and customize their web crawlers to achieve business goals.
The web crawler is basically a program that is mainly used for navigating to the web and finding new or updated pages for indexing. The crawler begins with a wide range of seed websites or popular URLs and searches depth and breadth to extract hyperlinks.
Famous search engines such as Google, Yahoo and Bing do web crawling and use this information for indexing web pages.
Try crawler4j. You just need to implement a simple interface which controls which URLs to visit and what to do with each crawled page.
119
in java I think it boils down to Nutch vs Heritrix. You should specify what your needs are to get a better answer.
34
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
