Scraping from a website that requires a login?

Question

Can this be done if so, how? I want to scrape data from xbox.com but the pages I need to scrape only appear after a successful login.

Answer 1

Most login forms will set a cookie. So you should use a HTTP class like Zend_Http that can store them for further requests. It's presumably as simple as:

$client = new Zend_Http_Client();
$client->setCookieJar();   // this is the crucial part for "logging in"

// make login request
$client->setUri("http://xbox.com/login");
$client->setParameterPost("login", "hackz0r");
$result = $client->request('POST');

// go scraping
...

Answer 2

You will have to go through the required login transaction by sending POST data with your CURL requests. That said, it is a bad idea to scrape data from behind a login - the site didn't put that information in the public for a reason, and for you to do so might constitute copyright infringement,