Selenium headless not working with custom profile selection

Question

Below is the code I'm using - I've attempted to make it as concise as possible.

import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

import bs4

options=Options()
#options.add_argument('--headless') # Works while not headless?
options.add_argument('--disable-gpu')  # Last I checked this was necessary.
options.add_argument("--user-data-dir=profiles\\") #Keeps login data
options.add_argument("--profile-directory=Profile 1")

driver=webdriver.Chrome(chrome_options=options)
driver.get("http://www.google.com")
html=driver.page_source
soup=bs4.BeautifulSoup(html, "html.parser")

print(soup)

The main problem stems from the --user-data-dir and --profile-directory parameter. In my test example a custom chrome profile (usually found at C:\Users\User\AppData\Local\Google\Chrome\User Data) is in the current directory to keep it separate from any currently running chrome sessions.

If the --headless parameter is enabled while using the above parameters the driver hangs (the CMD stays and does not produce output on the Python command line). However, when not enabled the window pops open and performs as expected.

However, --headless does work while using any profile within the above default directory.

This is the console output;

[0120/222514.611:ERROR:gpu_process_transport_factory.cc(967)] Lost UI shared context.

DevTools listening on ws://127.0.0.1:59961/devtools/browser/ee317ed6-93c7-47c2-b26d-63647980ba0d
[0120/222514.619:ERROR:devtools_http_handler.cc(289)] Error writing DevTools active port to file
[0120/222514.624:ERROR:cache_util_win.cc(19)] Unable to move the cache: 0
[0120/222514.625:ERROR:cache_util.cc(140)] Unable to move cache folder profiles\Default\GPUCache to profiles\Default\old_GPUCache_000
[0120/222514.625:ERROR:disk_cache.cc(184)] Unable to create cache
[0120/222514.625:ERROR:shader_disk_cache.cc(622)] Shader Cache Creation failed: -2

So, it looks like somewhere Chromium is assuming I'm using the Default profile when in fact I specify I am not.

Does anyone have any suggestions? Thanks in advance!

Answer 1

I was able to your code to run with the following option added.

options.add_argument('--remote-debugging-port=45447')

Without it the code was hanging for around a minute before the following error was thrown:

WebDriverException: Message: unknown error: DevToolsActivePort file doesn't exist

Comment excerpt which pointed me to a solution:

When you send --remote-debugging-port=0 to Chrome, devtools chooses its own port and writes it to chrome::DIR_USER_DATA as a file named "DevToolsActivePort".

Basically once the port is set to something other than the default of 0 the DevToolsActivePort file does not need to be checked. The full comment and bug are here: chromedriver bug

Hope this helps!