Phantom JS driver is unable to locate elements sometimes

Question

I am new to PhantomJS and I am trying to run my selenium tests (python) using phantomjs driver but It won't the web elements.

Ghostdriver logs:

[INFO  - 2015-02-27T15:24:40.236Z] GhostDriver - Main - running on port 52653
[INFO  - 2015-02-27T15:24:41.075Z] Session [bfd397f0-be94-11e4-ad03-b711254501c8] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1","webSecurityEnabled":true}
[INFO  - 2015-02-27T15:24:41.075Z] Session [bfd397f0-be94-11e4-ad03-b711254501c8] - page.customHeaders:  - {}
[INFO  - 2015-02-27T15:24:41.075Z] Session [bfd397f0-be94-11e4-ad03-b711254501c8] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"2.0.0","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"mac-10.9 (Mavericks)-64bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
[INFO  - 2015-02-27T15:24:41.075Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: bfd397f0-be94-11e4-ad03-b711254501c8
[ERROR - 2015-02-27T15:24:47.242Z] WebElementLocator - _handleLocateCommand - Element(s) NOT Found: GAVE UP. Search Stop Time: 1425050687190

  :262 in error

The intriguing part is that, after I successfully installed phantomjs, I ran my login test and it passed with no problem. Then I ran another test that failed for the same reason mentioned above. I tried to run again the Login test that passed - but the phantomjs driver would not find the elements anymore.

Any idea what is causing this?

By the way, these tests run fine with chrome and FF

Answer 1

So... it looks like any element that uses Selenium WebDriver Wait is actually using a polling scheme. Remember that a Selenium explicit wait is code you define to wait for a certain condition to occur before proceeding further in the code. WebDriverWait by default calls the ExpectedCondition every 500 milliseconds until it returns successfully. (reference.) What that means is every 500 mSecs, Selenium tests the wait condition. If true, proceed. If not yet true, wait for another poll frequency cycle, then try again.

And from my testing, my belief is that a poll test and not-ready-yet so call it a fail definitely generates an errors in my ghostdriver.log

[ERROR - 2016-08-14T08:50:12.896Z] WebElementLocator - _handleLocateCommand - Element(s) NOT Found: GAVE UP. Search Stop Time: 1471164612878

I'm working on a project that scrapes a complex single page AJAX site. Because the site re-uses common div elements, I have to make a whole lot of calls (clear, paste value, click, wait, go to a different table on page, copy data of interest, repeat...) to obtain the data that I need. And because the site is sort of slow, I apply wait elements via:

driver.wait.until(EC.presence_of_element_located((By.XPATH, "//*[@id='special_content_id']//td[contains(.,'" + info.unitId + "')]")))

It turns out that polling frequency is set in the class selenium.webdriver.support.wait.WebDriverWait(driver, timeout, poll_frequency=0.5, ignored_exceptions=None)

Reference here.

I was curious about the behavior. When my poll_frequency was set to 0.5 seconds, I received 98 WebElementLocator errors. When I switched that to 7.5 seconds, driver.wait = WebDriverWait(driver, 60, 7.5), I received only 36 errors. Smaller poll times generate more errors.

What I really find odd is that I wouldn't expect a normal polling behavior (Selenium) to set an log error anywhere (PhantomJS). I do wish there was a different log entry for a polling, element-not-yet-ready...