Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

What is the best Open Source Web Crawler Tool written in Java? [closed]

Tags:

java

web-crawler

What is the best Open Source Web Crawler Tool, written in Java.

like image 876
cuneytykaya Avatar asked Dec 12 '11 12:12

cuneytykaya

People also ask

Which web crawler is best?

Apache Nutch is unquestionably at the top of the web crawler tool heap when it comes to the greatest open source web crawlers. Apache Nutch is a prominent open source code web data extraction software project for data mining that is highly flexible and scalable.

What is an open source crawler?

What are open source crawlers? Web crawlers are a type of software that automatically targets online websites and pulls their data in a machine-readable format. Open source web crawlers enable users to: modify the code and customize their web crawlers to achieve business goals.

What is a Java web crawler?

The web crawler is basically a program that is mainly used for navigating to the web and finding new or updated pages for indexing. The crawler begins with a wide range of seed websites or popular URLs and searches depth and breadth to extract hyperlinks.

Is Google a web crawler or web scraper?

Famous search engines such as Google, Yahoo and Bing do web crawling and use this information for indexing web pages.

2 Answers

Try crawler4j. You just need to implement a simple interface which controls which URLs to visit and what to do with each crawled page.

like image 119
Andy Avatar answered Nov 09 '22 03:11

Andy

in java I think it boils down to Nutch vs Heritrix. You should specify what your needs are to get a better answer.

like image 34
riffraff Avatar answered Nov 09 '22 05:11

riffraff
Sign in to Comment

Related questions

How do we convert a String from PEM to DER format
How to open JFileChooser with predefined size
If JNI DeleteGlobalRef() is called, does the corresponding java object get garbage collected?
Java: complete list of @SuppressWarnings(...) parameters (in Netbeans)?
Byte array versus Base 64 string in RESTful web service
Google Guice: Provider with parameters
Android - use ant to create build configurations that change configuration values
AffineTransform without transforming Stroke?
What is the best practice to modify an object passed to a method
How to log non-string item in the slf4j
Is it clearer to always use "this" or only when necessary? [duplicate]
How to perform a complicated change of variables for a polynomial (in Mathematica)
ClassCircularityError thrown by ClassLoader.defineClass
Does mockito have an equivalent idiom to jMock's States?
How can I instantiate a generic type in Java?
Java JDBC - Multiple prepared statement bulk insert
Changing Border Color of AWT TextField
Does JVM/GC call `finalize()` on program/thread exit?
Mockito - Stubbing a method of an object that was returned by a mock object method
Why does compiling this code cause a compiler stack overflow?

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!