PhantomJS randomly does not exit (using Selenium with Python)

Question

I'm using selenium-python with PhantomJS. The code is pretty much like this:

from selenium.webdriver import PhantomJS
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
driver = PhantomJS()
wait = WebDriverWait(driver, 10)
driver.get(url)
while True:
    // scrap the page
    try:
        driver.find_elements_by_css_selector('.next')[0].click()
    except: break
    wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, '.loading')))
    wait.until(expected_conditions.invisibility_of_element_located((By.CSS_SELECTOR, '.loading')))
driver.quit()

I use a celery task which runs this code periodically. The problem is that from time to time there are some stale phantomjs processes. When I look into celery logs the task is completed successfully without any errors but the phantomjs process is still running.

Some extra info:

• I'm using Python 3.2
• I'm using Celery 3.1 with beat
• I'm using Debian Wheezy
• I compiled PhamtomJS from source and created a symbolic link like this: ln -s /opt/phantomjs/bin/phantomjs /usr/local/bin/

Can someone suggest a way to debug and find out who's fault is this?

Answer 1

I have not use celery before, are you sure celery will display all error message for you? For your code, I think if an exception occurs when running wait.until(...), driver.quit() will not be executed.