Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Very Simple C++ Web Crawler/Spider?

Tags:

I am trying to do a very simple web crawler/spider app in C++. I have been searching using Google for a simple one to understand the concept. I found this:

spider_simpleCrawler

However, it is complicated to understand for me, since I started learning C++ about 1 month ago.

This is, for example, what I'm trying to do:

  1. Enter the URL: www.example.com (I will use bash->wget, to get the contents/source code),

  2. Look for, maybe "a href" link, and then store in some data file.

Is there a simpler tutorial or guide on the Internet?

like image 824
popurity09 Avatar asked Nov 25 '10 14:11

popurity09


People also ask

What is simple web crawler?

A web crawler, or spider, is a type of bot that is typically operated by search engines like Google and Bing. Their purpose is to index the content of websites all across the Internet so that those websites can appear in search engine results.

How do I make a web crawler?

Here are the basic steps to build a crawler: Step 1: Add one or several URLs to be visited. Step 2: Pop a link from the URLs to be visited and add it to the Visited URLs thread. Step 3: Fetch the page's content and scrape the data you're interested in with the ScrapingBot API.


2 Answers

All right, I'll try to point you in the right direction. Conceptually, a webcrawler is pretty simple. It revolves around a FIFO queue data structure which stores pending URLs. C++ has a built-in queue structure in the standard libary, std::queue, which you can use to store URLs as strings.

The basic algorithm is pretty straightforward:

  1. Begin with a base URL that you select, and place it on the top of your queue
  2. Pop the URL at the top of the queue and download it
  3. Parse the downloaded HTML file and extract all links
  4. Insert each extracted link into the queue
  5. Goto step 2, or stop once you reach some specified limit

Now, I said that a webcrawler is conceptually simple, but implementing it is not so simple. As you can see from the above algorithm, you'll need: an HTTP networking library to allow you to download URLs, and a good HTML parser that will let you extract links. You mentioned you could use wget to download pages. That simplifies things somewhat, but you still need to actually parse the downloaded HTML docs. Parsing HTML correctly is a non-trivial task. A simple string search for <a href= will only work sometimes. However, if this is just a toy program that you're using to familiarize yourself with C++, a simple string search may suffice for your purposes. Otherwise, you need to use a serious HTML parsing library.

There are also other considerations you need to take into account when writing a webcrawler, such as politeness. People will be pissed and possibly ban your IP if you attempt to download too many pages, too quickly, from the same host. So you may need to implement some sort of policy where your webcrawler waits for a short period before downloading each site. You also need some mechanism to avoid downloading the same URL again, obey the robots exclusion protocol, avoid crawler traps, etc... All these details add up to make actually implementing a robust webcrawler not such a simple thing.

That said, I agree with larsmans in the comments. A webcrawler isn't the greatest way to learn C++. Also, C++ isn't the greatest language to write a webcrawler in. The raw-performance and low-level access you get in C++ is useless when writing a program like a webcrawler, which spends most of its time waiting for URLs to resolve and download. A higher-level scripting language like Python or something is better suited for this task, in my opinion.

like image 73
Charles Salvia Avatar answered Oct 03 '22 21:10

Charles Salvia


Check this Web crawler and indexer written in C++ at: Mitza web crawler The code can be used as reference. Is clean and provides good start for a webcrawler codding. Sequence diagrams can be found at the above link pages.

like image 25
user2195463 Avatar answered Oct 03 '22 22:10

user2195463