How to know if the website being scraped has changed?

Question

I'm using PHP to scrape a website and collect some data. It's all done without using regex. I'm using php's explode() method to find particular HTML tags instead.

It is possible that if the structure of the website changes (CSS, HTML), then wrong data may be collected by the scraper. So the question is - how do I know if the HTML structure has changed? How to identify this before storing any data to my database to avoid wrong data being stored.

Answer 1

I think you don't have any clean solutions if you are scraping a page where content changes.

I have developed several python scrapers and I know how can be frustrating when site just makes a subtle change on its layout.

You could try a solution a la mechanize (don't know the php counterpart) and if you are lucky you could isolate the content you need to extract (links?).

Another possibile approach would be to code some constraints and check them before store to db.

For example, if you are scraping Urls, you will need to verify that what scraper has parsed is formally a valid Url; same for integer ID or whatever you want to scrape that can be recognized as valid.

If you are scraping plain text, it will be more difficult to check.

Answer 2

If you want to know changes with respect to structure, I think the best way is to store the DOM structure of your first page and then compare it with new one.

There are lot of way you can do it:- SaxParser DOmParser etc

I have a small blog which will give some pointers to what I mean http://let-them-c.blogspot.com/2009/04/xml-as-objects-in-oops.html

or you can use http://en.wikipedia.org/wiki/Simple_API_for_XML or DOm Utility parser.

Answer 3

Depends on the site but you could count the number of page elements in the scraped page like div, class & style tags then by comparing these totals against those of later scrapes detect if the page structure has been changed.

A similiar process could be used for the CSS file where the names of each each class or id could be extracted using simple regex, stored and checked as needed. If this list has new additions then the page structure has almost certainly changed somewhere on the site being scraped.

Answer 4

Speaking out of my ass here, but its possible you might want to look at some Document Object Model PHP methods.

http://php.net/manual/en/book.dom.php

If my very, very limited understanding of DOM is correct, a change in HTML site structure would change the Document Object Model, but a simple content change within a fixed structure wouldn't. So, if you could capture the DOM state, and then compare it at each scrape, couldn't you in theory determine that such a change has been made?

(By the way, the way I did this when I was trying to get an email notification when the bar exam results were posted on a particular page was just compare file_get_contents() values. Surprisingly, worked flawlessly: No false positives, and emailed me as soon as the site posted the content.)