Conflicting solutions for different errors with Python/Selenium/Chrome

Question

I have been getting a series of pesky Selenium/Chrome errors for a week or so, where everything works fine for a while and then suddenly doesn't. I have the right version of Chromedriver (89.0.4389.23) for my version of Google Chrome (89.0.4389.114) per Google Chrome official docs

Currently I'm getting the following error when I run driver = webdriver.Chrome(options=options) (at least on my production server, I don't get it for the same code repo on my local machine):

selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
  (Session info: headless chrome=89.0.4389.114)

Frustratingly, the only solution I've found for this error aside from "make sure your chrome and chromedriver match" is to remove options.add_argument("--remote-debugging-port=9222") from my webdriver options. I do have that line of code as well, and removing it appears to solve my problem for now, but my problem that line of code appears under the following comments in my codebase:

    # adding arg below fixed "DevToolsActivePort file doesn't exist."
    # ht https://github.com/heroku/heroku-buildpack-google-chrome/issues/46
    options.add_argument("--remote-debugging-port=9222")

So I've now reached a perfect circle where my Python Selenium code reliably works until it reliably breaks, with either of two different Selenium errors ("session not created from disconnected: unable to connect to renderer" or "DevToolsActivePort file doesn't exist."), where the solution to one is to add a specific line of code and the solution to the other is to remove it.

Below is my current launch_webdriver function, with comments reflecting the various transient Chrome/Selenium errors I've been trying to navigate:

from selenium import webdriver
import chromedriver_binary
from selenium.webdriver.chrome.options import Options

def launch_browser(headless=False):  # :, driverpath="./spinocchio/chromedriver"):
    options = Options()

    # if any other driver `options` are added, they must be added BELOW no-sandbox
    # per user parsecer's comment at https://stackoverflow.com/a/53073789/1870832
    # and my own excruciating experience.
    options.add_argument("--no-sandbox")

    # you may think the two lines below can be replaced with just options.headless=headless
    # please don't.
    if headless:
        options.add_argument("headless")

    # adding arg below fixed "DevToolsActivePort file doesn't exist."
    # ht https://github.com/heroku/heroku-buildpack-google-chrome/issues/46
    options.add_argument("--remote-debugging-port=9222")
    driver = webdriver.Chrome(options=options)
    return driver

How can I get to something more reliable/robust?

Answer 1

Overview

1. Prevalent issue - per @ss7777
2. Albeit numerous potential causes for this/related

Blue-skies and 'silver-bullet' solution I'm afraid, however, here are some common causes and potential resolve / considerations etc.

---

Fixes/resolve

(accompanied by subj. view of 'commonality' | frequency)

1. Driver: ChromeDriver version/file location (~50%) - incorrect version + ensure location to executable correct location

"This issue is pretty evident when there is a incompatibility between the version of the binaries you are using." (adapted from this source, with useful summary of historical versions v/ binaries)

2. Chrome/driver still 'active' 5-20% - despite closing down, process tree elements can remain 'alive' or 'spinning' for some time - do you notice this right after restart of PC (here)?

(Have suffered and even evidenced this previously - e.g. running code in parallel, or simply executing through .bat at same time as PyCharm console).

3. Updates: plugin/other incompatibly (~5-15%) - noting you're using headless - does it work with head?)
4. System: clean up your system/restore-point or worst-face (~10-20%). (restore point worked for me)
5. Other: spawn separate thread (~5% - see here)/alternative ports (as req/feasible)

---

Workarounds

• Chrome dependency (e.g. see here - albeit potential red-herring..)
• Mode of execution (through editor main program, through CMD interface hosted via editor (e.g. PyCharm console), or through .bat file (as mentioned previously), Standalone Selenium here
• Configuration (technical/advanced - e.g. here)for advanced users)

---

Unresolved (albeit with promise)

Several cases of where such a Q remains unresolved/limited contribution:

• Unable to receive message from render here
• SessionNotCreatedException: Message: session not created from disconnected: unable to connect to renderer with ChromeDriver 2.45 Chrome v71

(although 2nd point here points to remedy in item 2 above).

---

Extreme solns:

• Have seen intense investigation and many too-fro re Q&A - with ultimate conclusion being complete reinstallation (Chrome/python, editor) etc.

---

Links / related