Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using Web crawler for price comparison

I need a open source java based web crwaler which I can extend for price comparison? How do I do the price comparison? Is there any open source code for that?

like image 593
yeskay Avatar asked Feb 16 '10 00:02

yeskay


2 Answers

Take a look at web harvest, you will have to use it's slightly odd and peculiar syntax for processing web pages, but it should be fairly to extend it to do some price comparison:

http://web-harvest.sourceforge.net/samples.php?num=2

like image 170
Jon Avatar answered Oct 11 '22 02:10

Jon


Building something that scrapes price information from a large number of different sites is going to be a lot of work, whether you scrape from the stores themselves or from existing comparison sites.

  • Everyone's website layout will be different, requiring you to configure your crawler separately for each one.

  • Some websites may present the price information in ways that make scraping difficult; e.g. using AJAX.

Some website owners will put the relevant pages into their robots.txt files to tell you to stay away. And if you ignore that, there are various things they can do to make life difficult for you.

Scraping lots of people's websites without permission is likely to make you unpopular. It might attract threats of lawsuits, or actual lawsuits from people who perceive that you are harming their business model. Or other responses ...

Are you really sure you want to do this? Really??

like image 23
Stephen C Avatar answered Oct 11 '22 03:10

Stephen C