unknown error: failed to wait for extension background page to load: chrome-extension error loading an extension to Chrome Headless using Selenium

Question

I try to run chromedriver via selenium in headless mode.

IMPORTANT The code runs perfectly fine if I eliminate the following code lines (but is not headless):

chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

This is the error I get when I try to implement the headless argument:

Traceback (most recent call last):
  File "camel.py", line 83, in <module>
    executable_path=executable_path)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to wait for extension background page to load: chrome-extension://jkompbllimaoekaogchhkmkdogpkhojg/_generated_background_page.html
from unknown error: page could not be found: chrome-extension://jkompbllimaoekaogchhkmkdogpkhojg/_generated_background_page.html

This are the lines 81, 82 and 83

chrome_options.add_extension(extension_path)
driver = webdriver.Chrome(options=chrome_options,
                          executable_path=executable_path)

This is the code (the crhomedriver execution parts):

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.expected_conditions import presence_of_element_located

import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
log_path = os.path.join(BASE_DIR, 'cronJobChromeDriver.log')

executable_path = os.path.join(BASE_DIR, 'chromedriver_linux64/chromedriver')
extension_path = os.path.join(
    BASE_DIR, 'chromedriver_linux64/extension_2_8_9_0.crx')
print('executable_path', executable_path)

Answer 1

The bottom line is, No, google-chrome-headless doesn't supports extensions.

In the one of his comment, [email protected] mentioned:

I realize a lot of folks would like to use extensions with headless but unfortunately that's a large project which we have /no plans to do/. The problem is Headless Chromium is a content embedder which means it doesn't have access to anything from other content embedders such as chrome and unfortunately extensions are a chrome feature.

In another comment he further added, if you're using Selenium through DevTools you can build a proxy. Next you can filter URLs and modify headers via Network.setRequestInterception and Network.continueInterceptedRequest.

---

Reference

You can find a relevant detailed discussion in:

• Is it possible to run Google Chrome in headless mode with extensions?

Answer 2

this is now possible by modifying the following flag:

chrome_options.add_argument('--headless=chrome')

I tested it successfully.

I found it here: https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c5