Read javascript code in selenium without opening browser

Question

I'm working with selenium API to webcrapping on pages with javascript.

Is there some method to get the code without a web browser screen opens ?

I am new to this API

Is possible?

Answer 1

You have, at least, 3 basic options:

• use a headless browser, like PhantomJS, example:

  >>> from selenium import webdriver
  >>> driver = webdriver.PhantomJS()
  >>> driver.get('http://stackoverflow.com')
  >>> driver.title
  u'Stack Overflow'
  

• use a virtual display (see xvfb) with the help of pyvirtualdisplay, examples here:

  • How do I run Selenium in Xvfb?
  • Selenium with pyvirtualdisplay unable to locate element

• use a remote selenium server, either your own with setting up own nodes in a grid, or at, for example, BrowserStack, or Sauce Labs:

  >>> from selenium import webdriver
  >>> from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
  >>> 
  >>> desired_cap = {'os': 'Windows', 'os_version': 'xp', 'browser': 'IE', 'browser_version': '7.0' }
  >>> driver = webdriver.Remote(command_executor='http://username:[email protected]:80/wd/hub', desired_capabilities=desired_cap)
  >>> 
  >>> driver.get('http://stackoverflow.com')
  >>> driver.title
  u'Stack Overflow'
  

Answer 2

Unfortunately, you can't use JavaScript without an interpreter, which is in a browser. Though, you may use PhantomJS - a headless browser.