Selenium leaves behind running processes?

Question

When my selenium program crashes due to some error, it seems to leave behind running processes.

For example, here is my process list:

carol    30186  0.0  0.0 103576  7196 pts/11   Sl   00:45   0:00 /home/carol/test/chromedriver --port=51789
carol    30322  0.0  0.0 102552  7160 pts/11   Sl   00:45   0:00 /home/carol/test/chromedriver --port=33409
carol    30543  0.0  0.0 102552  7104 pts/11   Sl   00:48   0:00 /home/carol/test/chromedriver --port=42567
carol    30698  0.0  0.0 102552  7236 pts/11   Sl   00:50   0:00 /home/carol/test/chromedriver --port=46590
carol    30938  0.0  0.0 102552  7496 pts/11   Sl   00:55   0:00 /home/carol/test/chromedriver --port=51930
carol    31546  0.0  0.0 102552  7376 pts/11   Sl   01:16   0:00 /home/carol/test/chromedriver --port=53077
carol    31549  0.5  0.0      0     0 pts/11   Z    01:16   0:03 [chrome] <defunct>
carol    31738  0.0  0.0 102552  7388 pts/11   Sl   01:17   0:00 /home/carol/test/chromedriver --port=55414
carol    31741  0.3  0.0      0     0 pts/11   Z    01:17   0:02 [chrome] <defunct>
carol    31903  0.0  0.0 102552  7368 pts/11   Sl   01:19   0:00 /home/carol/test/chromedriver --port=54205
carol    31906  0.6  0.0      0     0 pts/11   Z    01:19   0:03 [chrome] <defunct>
carol    32083  0.0  0.0 102552  7292 pts/11   Sl   01:20   0:00 /home/carol/test/chromedriver --port=39083
carol    32440  0.0  0.0 102552  7412 pts/11   Sl   01:24   0:00 /home/carol/test/chromedriver --port=34326
carol    32443  1.7  0.0      0     0 pts/11   Z    01:24   0:03 [chrome] <defunct>
carol    32691  0.1  0.0 102552  7360 pts/11   Sl   01:26   0:00 /home/carol/test/chromedriver --port=36369
carol    32695  2.8  0.0      0     0 pts/11   Z    01:26   0:02 [chrome] <defunct>

Here is my code:

from selenium import webdriver

browser = webdriver.Chrome("path/to/chromedriver")
browser.get("http://stackoverflow.com")
browser.find_element_by_id('...').click()

browser.close()

Sometimes, the browser doesn't load the webpage elements quickly enough so Selenium crashes when it tries to click on something it didn't find. Other times it works fine.

This is a simple example for simplicity sake, but with a more complex selenium program, what is a guaranteed clean way of exiting and not leave behind running processes? It should cleanly exit on an unexpected crash and on a successful run.

Answer 1

I see this pretty old thread, but maybe my case will be useful for somebody. For some reasons I had to run a lot of scrapers with separate webdriver instance with headfull (non headless) browser for every request in Docker container with Xvfb. So every request produced 2-3 zombie processes with the firefox. (and 12 whith Chromedriver). So after few minutes of scraing I had thousands of zombie processes. driver.close() and driver.quit() had no success. The Jimmy's Engelbrecht solution is better, but it was killing only part of processes. So the only working method for me was to enable init in docker container.

docker run --init container

It protects you from software that accidentally creates zombie processes, which can (over time!) starve your entire system for PIDs (and make it unusable).

• If you use a docker-compose you can enable this as described in this answer
• For the Docker < 1.13 you can implement https://github.com/krallin/tini manually.

Answer 2

As already pointed out you should run browser.quit()

But on linux (inside docker) this will leave defunct processes. These are typically not really a problem as they are mere an entry in the process-table and consume no resources. But if you have many of those you will run out of processes. Typically my server melts down at 65k processes.

It looks like this:

# root@dockerhost1:~/odi/docker/bf1# ps -ef | grep -i defunct | wc -l
28599

root@dockerhost1:~/odi/docker/bf1# ps -ef | grep -i defunct | tail
root     32757 10839  0 Oct18 ?        00:00:00 [chrome] <defunct>
root     32758   895  0 Oct18 ?        00:00:02 [chrome] <defunct>
root     32759 15393  0 Oct18 ?        00:00:00 [chrome] <defunct>
root     32760 13849  0 01:23 ?        00:00:00 [chrome] <defunct>
root     32761   472  0 Oct18 ?        00:00:00 [chrome] <defunct>
root     32762 19360  0 01:35 ?        00:00:00 [chrome] <defunct>
root     32763 30701  0 00:34 ?        00:00:00 [chrome] <defunct>
root     32764 17556  0 Oct18 ?        00:00:00 [chrome] <defunct>
root     32766  8102  0 00:49 ?        00:00:00 [cat] <defunct>
root     32767  9490  0 Oct18 ?        00:00:00 [chrome] <defunct>

The following code will solve the problem:

def quit_driver_and_reap_children(driver):
    log.debug('Quitting session: %s' % driver.session_id)
    driver.quit()
    try:
        pid = True
        while pid:
            pid = os.waitpid(-1, os.WNOHANG)
            log.debug("Reaped child: %s" % str(pid))

            #Wonka's Solution to avoid infinite loop cause pid value -> (0, 0)
            try:
                if pid[0] == 0:
                    pid = False
            except:
                pass
            #---- ----

    except ChildProcessError:
        pass