Chrome Webdriver produces timeout in Selenium

Question

I am trying to scrape a website using Selenium and Chrome's webdriver and this all worked fine until I switched to a newer Macbook. All of sudden, the webdriver seems to not recognize when the website is actually fully loaded.

Error message goes as follows

TimeoutException: Message: timeout: cannot determine loading status from timeout: Timed out receiving message from renderer: -0.003
(Session info: chrome=54.0.2840.87) (Driver info: chromedriver=2.25.426935 (820a95b0b81d33e42712f9198c215f703412e1a1),platform=Mac OS X 10.12.1 x86_64)

My code looks like this:

      import os
      import time
      from selenium import webdriver
      driver = webdriver.Chrome(os.path.join(os.getcwd(), 'chromedriver'))
      driver.get('http://www.clever-tanken.de/')

Answer 1

This is a known bug in the chrome webdriver. Ex1, Ex2

After a very brief read, it looks like their developers are having some trouble reproducing the bug. @dimkin mentioned CDNs as a possible cause for the bug. It looks like others have reported the bug and have similar suspicions that cdn content is not resolving DNS appropriately.

I was able to isolate the issue. In my case it was happening only for pages referencing a static file (an image in a HTML tag for instance http://cdn.local.myproject.net/static/myimage.png) on my custom local cdn domain. The was not present if I used a relative path "/static/myimage.png" or localhost "http://127.0.0.1/static/myimage.png" so I figured it was a DNS problem.

I was able to bypass the problem by using the --dns-prefetch-disable option of chrome.

Example:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--dns-prefetch-disable")
driver = webdriver.Chrome(chrome_options=chrome_options)

Another workaround mentioned looks promising. It doesn't fix anything but could possibly get around the problem. It catches the exception and simply attempts a refresh:

You'll need: from selenium.common.exceptions import TimeoutException

try:
   webDriver.get(url);
except TimeoutException as ex:
   print(ex.Message)
   webDriver.navigate().refresh()