How can I get the HTML generated with javascript?

Question

I want to get the HTML content of a web page but most of the content is generated by javascript.

Is it posible to get this generated HTML (with python if posible)?

Answer 1

The only way I know of to do this from your server is to run the page in an actual browser engine that will parse the HTML, build the normal DOM environment, run the javascript in the page and then reach into that DOM engine and get the innerHTML from the body tag.

This could be done by firing up Chrome with the appropriate URL from Python and then using a Chrome plugin to fetch the dynamically generated HTML after the page was done initializing itself and communicate back to your Python.

Answer 2

Checkout Selenium. It have a python driver, which might be what you're looking for.