Python Selenium Enable Javascript Issue

Question

I am trying to scrape some info from a website using python selenium. However, when I submit the form, the result page keeps showing me "Your web browser requires JavaScript to access the page". Can anyone let me know how to solve this?

Please see below for my code:

from selenium import webdriver

profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0")
profile.set_preference("javascript.enabled", True)
broswer = webdriver.Firefox(profile)
broswer.get(http://www.cathaypacific.com/cx/en_CA.html)

input = broswer.find_element_by_id('depart-label')
input.clear()
input.send_keys('Hong')

WebDriverWait(broswer, 10, poll_frequency=0.1).until(lambda drv:    len(drv.find_elements_by_css_selector("ul.ui-autocomplete li")) > 0)
broswer.find_element_by_css_selector("ul.ui-autocomplete li").click()

time.sleep(3)
input = broswer.find_element_by_id('destination-label')
input.send_keys('van')
WebDriverWait(broswer, 10, poll_frequency=0.1).until(lambda drv: len(drv.find_elements_by_css_selector("ul.ui-autocomplete li")) > 0)
broswer.find_element_by_css_selector("#ui-id-2 li").click()

broswer.find_element_by_class_name("button-submit").click()

Answer 1

There is nothing wrong with the code itself. It worked for me for the first time.

Now, the real problem is that this particular website is using a third-party anti web-scraping service called Distil Networks, which apparently has a way of detecting a selenium-webdriver powered browser, more information here:

• Can a website detect when you are using selenium with chromedriver?