Unable to locate elements on webpage with headless chrome

Question

I have a script that's accessing printers, and my code works totally fine when chrome is run normally, but when it's run headless, selenium can't seem to find elements on the webpage.

Here's the relevant code:

init method:

def __init__(self, ip_address):
    """ Initialize a new Printer_Webpage object."""
    self.ip_address = ip_address
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--window-size=1920x1080")
    self.browser = webdriver.Chrome(chrome_options=chrome_options)
    # Ignore lack of cert for each printer web page.
    # Otherwise, can't open page.
    self.browser.accept_untrusted_certs = True

Login method:

def login(self):
    """Navigates through the login page for the printer."""
    # Open login page
    self.browser.get(f'https://{self.ip_address}/wcd/top.xml')
    # STEPS TO LOGIN:
    # 1) Select 'Administrator' radio button and click.
    self.browser.find_element_by_id('Admin').click()
    # 2) Select Login button and click.
    self.browser.find_element_by_xpath("//input[@type='submit' \
                                        and @value='Login']").click()
    # 3) Select admin (user mode)
    self.browser.find_element_by_id('R_ADM2').click()
    # 4) Select password field and input PASSWORD, then submit.
    password_field = self.browser.find_element_by_id('Admin_Pass')
    password_field.send_keys(PASSWORD)
    password_field.send_keys(Keys.RETURN)

Full error message:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"Admin"}

And here's some other info that might be of use:

(Session info: headless chrome=62.0.3202.94)

(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 10.0.14393 x86_64)

Answer 1

I had the same problem. You could take screenshots to understand whats wrong.

driver.get_screenshot_as_file("screenshot.png")

A few reasons why selenium works when run normally but stops working in headless mode -

1)It might have switched to a mobile template. Can be fixed by changing the window size.

chrome_options.add_argument("--window-size=1920,1080")

2)If its a blank page (screenshot), it might be due to an invalid SSL certificate.(see @Marcel_Wilson post) It should be fixed by -

chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-running-insecure-content')

3)Its possible that the website blocks 'headless' mode. (Your screenshots might show errors which you cannot recreate in normal mode) You could try this-

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36'
options.add_argument(f'user-agent={user_agent}')

However, the above code won't work if the website has a more robust blocking method. You can find more about this here https://intoli.com/blog/making-chrome-headless-undetectable/.

Answer 2

If it's an issue with SSL certs, you can start Chrome without certs using a command line flag (assuming that's how you're starting it). I believe the switch is --allow-running-insecure-content, and I located that from this list found here.