Selenium Chrome Webdriver not working in headless mode with profile

Question

So, this is the code I'm having troubles with:

def scrap():
        options = webdriver.ChromeOptions();
        options.add_argument('headless');
        options.add_argument('--profile-directory=Profile 1')
        options.add_argument("--user-data-dir=C:/Users/omarl/AppData/Local/Google/Chrome/User Data/")
        options.add_argument("--remote-debugging-port=45447")
    
        options.add_argument("--disable-gpu") 
        browser = webdriver.Chrome(executable_path=r"C:\Users\omarl\OneDrive\Escritorio\chromedriver.exe", options=options)
        
        scrapURL = "https://es.wallapop.com/search?distance=30000&keywords=leggins&latitude=41.38804&longitude=2.17001&filters_source=quick_filters"
        browser.get(scrapURL)
        #...

And the error:

WebDriverException: unknown error: unable to discover open pages

I don't have any instances of chrome when I execute the script, and when I'm using it without the headless option it works fine. Any idea why this is happening? Please, note that I'm using the --remote-debuggin-port provided in similar questions.

I'm using ChromeDriver 86.0.4240.22

Answer 1

To invoke a Chrome Profile in Headless mode you can use the --user-data-dir argument only and you can safely remove --profile-directory argument as follows:

• Code Block:

  from selenium import webdriver
  
  options = webdriver.ChromeOptions() 
  options.add_argument('--headless')
  options.add_argument('--window-size=1920,1080')
  # options.add_argument('--profile-directory=Profile 1')
  options.add_argument(r"--user-data-dir=C:\Users\Soma Bhattacharjee\AppData\Local\Google\Chrome\User Data\Default")
  options.add_argument("--remote-debugging-port=9222")
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get('https://www.google.com/')
  print("Chrome Headless launched")
  

• Console Output:

  DevTools listening on ws://127.0.0.1:9222/devtools/browser/93c67c41-e125-4d12-abc0-fcf0f07a62f4
  Chrome Headless launched
  

---

References

You can find a couple of relevant detailed discussions in:

• How to open a Chrome Profile through --user-data-dir argument of Selenium
• Selenium: Point towards default Chrome session

---

Additional Considerations

Ensure that:

• Selenium is upgraded to current released Version 3.141.0.
• ChromeDriver is updated to current ChromeDriver v86.0 level.
• Chrome is updated to current Chrome Version 86.0 level. (as per ChromeDriver v86.0 release notes).
• Execute your @Test as non-root user.
• Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.

---

tl; dr

ChromeDriver remote debug port reservation race conditions