Selenium webdriver driver.get(url) hangs... sometimes

Question

I'm using PhantomJS as a webdriver to load some urls. Usually, the program runs fine. However, it hangs on driver.get(url) a lot, and i'm wondering if there is anything I can do about it?

driver = webdriver.PhantomJS(executable_path= path_to_phantomjs_exe, service_log_path= path_to_ghostdriver_log)
driver.get(url)

It will just hang trying to load a certain url forever. But if i try it again, it might work. Are webdrivers/phantomJS really just that unstable? I guess last resort would be to constantly call driver.get(url) until it finally loads, but is that really going to be necessary? Thanks!

EDIT: It seems to only hang when loading the first link out of a list of them. It eventually does load, however, but after a few minutes. The rest of the links load within seconds. Any help at all would be great.

Answer 1

I've answered this exact problem on this post here: Geb/Selenium tests hang loading new page but copied it here because I see that this question is older.

I hope you can find a way to implement this into your code, but this is what worked for me when I was having a similar situation with PhantomJS hanging.

I traced it to be hanging on a driver.get() call, which for me was saying something wasn't going through or the webdriver just wasn't - for some reason - giving the load successful command back to the driver, allowing the script to continue.

So, I added the following:

driver = webdriver.PhantomJS()

# set timeout information
driver.set_page_load_timeout(15)

I've tested this at a time of 5 (seconds) and it just didn't wait long enough and nothing would happen. 15 seconds worked great for me, but that's maybe something you should test.

On top of this, I also created a loop whenever there was an option for the webdriver to timeout, so that the driver.get() could attempt to re-send the .get() command. Implementing a try / except stacked scenario, I was able to approach this:

while finished == 0:
    try:
        driver.get(url3)
        finished = 1
    except:
        sleep(5)

I have seen an except handle as:

except TimeoutException as e:
    #Handle your exception here
    print(e)

but I had no use for this. It might be nice to know how to catch specific exceptions, though.

See this solution for more options for a timeout: Setting timeout on selenium webdriver.PhantomJS