Get page generated with Javascript in Python

Question

I'd like to download web page generated by Javascript and store it to string variable in Python code. The page is generated when you click on button.

If I would know the resulting URL I would use urllib2 but this is not the case.

thank you

Answer 1

You could use Selenium Webdriver:

#!/usr/bin/env python from contextlib import closing from selenium.webdriver import Firefox # pip install selenium from selenium.webdriver.support.ui import WebDriverWait  # use firefox to get page with javascript generated content with closing(Firefox()) as browser:      browser.get(url)      button = browser.find_element_by_name('button')      button.click()      # wait for the page to load      WebDriverWait(browser, timeout=10).until(          lambda x: x.find_element_by_id('someId_that_must_be_on_new_page'))      # store it to string variable      page_source = browser.page_source print(page_source)