how to scrape html generated by javascript using python?

Question

I want to scrape the html generated by javascript , just like what you can see in Firebug.

UPDATE: I've found this article: http://blog.motane.lu/2009/07/07/downloading-a-pages-content-with-python-and-webkit/ which use PyQt to solve the problem and it works well for me.

BUT another problem occur: I have to login the website first, but I don't know how to simulate login in PyQt .... :(

Answer 1

Have a look at this article which describes using Windmill to do scrape a page after Javascript has been executed by the browser.

This article will show how to extract the desired information using the same three steps when the web page is not written directly using HTML, but is auto-generated using JavaScript to update the DOM tree.

They have some examples I am sure you can easily adapt.