How to make mechanize wait for web-page 'full' load?

Question

I want to scrape some web page which loads its components dynamically. This page has an onload script, and I can see the complete page 3-5 seconds after typing the URL into my browser.

The problem is, when I call br.open('URL'), the response is the web page at 0 seconds. There is a difference 3-5 seconds later between the HTML (which I want) and result of br.open('URL').

Answer 1

The problem you're having is that the web page is rendered in your web browser through the javascript engine. However, mechanize doesn't have the ability to execute javascript on its own so, no matter how long you wait, you aren't going to get the HTML you're missing using just mechanize.

For more information about how scrape dynamically generated content, please have a look at this question.

Answer 2

Working a webpage with a rich javascripts content with mechanize is not much easy, but there are ways to get what you want according to different situations.

• If some json requests are made to create the content, then you can call that urls and try to parse responses to get content, then try to join it properly.

• If you need to use some forms, you can create some form fields and set their values within mechanize. Or , simply write a method that will encode your POST or GET data (quote special characters etc..) and send them with mechanize.browser.open method.

• If page has some javascript based security functions (like some special encoding to form data before posting them), then you may use node.js like javascript application servers to process some javascript code blocks.

But in fact, some of the above options are not easy to do, and you must think twice before using mechanize for such projects.