python selenium does not scale with firefox - Message: Can't load the profile. Profile Dir

Question

Why does this happen using selenium with python and firefox? Selenium cant scale to a simple loop. Why cant firefox scale? Over time it decides to thow an error and quit.

Mozilla Firefox 45.0 selenium.version '2.53.2'

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
mylist = ['page1','page1',...,'pageN']
while True:
   for data in mylist:

        browser = webdriver.Firefox()
        try:
           myfunc(data) # Code that uses selenum to fetch pages
        except:
            pass
        browser.quit()
   time.sleep(60)



No handlers could be found for logger "sentry.errors.serializer"
Traceback (most recent call last):
  File "platrieveerp.py", line 231, in <module>
    browser = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/webdriver.py", line 103, in __init__
    self.binary, timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable
    % (self.profile.path))
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpOgsKOx If you specified a log_file in the FirefoxBinary constructor, check it for details.

Answer 1

Selenium is not failing while running the loop, it is failing at webdriver.Firefox(). The error is recognized when the connection times out at a certain time after line 231 when the script still cant recognize an open firefox application.

It is just a bug in selenium when running with a headless browser. To get a working version of selenium, run

sudo pip install -U selenium 

This should be a similar issue: Selenium: FirefoxProfile exception Can't load the profile