Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Selenium working with Chrome, but not headless Chrome

I've developed a couple of Python scripts using Selenium and, at first, PhantomJS. While heading toward automated downloads, I switched for (headed) Firefox (which worked) and then Chrome with the headless option so I won't have the browser opening in front of me.

My first script, which accesses a page and a couple of HTML elements, works perfectly with headless Chrome.

The second one, however, works only with headed Chrome. If I add the "headless" option, it doesn't work anymore. When I try to print the HTML in headless mode to see why it cannot find the HTML element I'm looking for, all I have is :

<html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>

With the headed Chrome, I have a complete HTML printed. This is how I start my headless Chrome :

options = webdriver.ChromeOptions()
options.add_argument("--ignore-certificate-errors") 
options.add_argument("headless") 
driver = webdriver.Chrome(chrome_options=options)

Again, note that this works in another of my script. The only difference here is that I need to log in to access the page, but even then, why would it work with the head? My script is made to log in automatically anyway by filling the form.

Python : 3.6.1, Chrome : 60.0.3112.78 (64 bits), Selenium : 3.4.3

Any idea? Thanks.

** EDIT: Here is the beginning of the code**

url = 'https://10.11.227.21/tmui/'
driver.get(url + "login.jsp")

html_source = driver.page_source
print(html_source)

blocStatus = WebDriverWait(driver, TIMEOUT).until(EC.presence_of_element_located((By.ID, "username")))
inputElement = driver.find_element_by_id("username")
inputElement.send_keys('actualLogin')
inputElement = driver.find_element_by_id("passwd")
inputElement.send_keys('actualPassword')
inputElement.submit()
like image 809
Gnougnou Avatar asked Jul 28 '17 09:07

Gnougnou


People also ask

Does Selenium use headless Chrome?

Luckily, you can configure Selenium to run the Chrome web browser in headless mode. In this mode, the Chrome web browser will run without any graphical user interface. So, Selenium can do web automation, web scrapping, browser tests, etc.

Does Selenium support headless browser?

Selenium supports headless browser testing using HtmlUnitDriver. HtmlUnitDriver is based on java framework HtmlUnit and is the one of the lightweight and fastest among all headless browser.

How do I enable headless in Chrome?

After this thing, we have to do is to create a WebDriver object, and set the ChromeDriver path and some arguments : ChromeOptions options = new ChromeOptions();options. addArguments("--headless");WebDriver driver = new ChromeDriver(options); That is all.

How do I run Chromedriver in headless mode?

Mastering XPath and CSS Selector for Selenium Post version 59, Chrome supports headless execution. ChromeOptions class is utilized to modify the default characteristics of the browser. The addArguments method of the ChromeOptions class is used for headless execution and headless is passed as a parameter to that method.


4 Answers

I had a same experience like you, and solved it by using xvfb and pyvirtualdisplay.

I use chromedrive=v2.3.1, chrome-browser=v60 and Selenium=3.4.3

In Headless chrome, some of script seems not to work as expected.

Please refer to vpassapera's comment in https://gist.github.com/addyosmani/5336747.

How about try it like below,

from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()

# Do Not use headless chrome option
# options.add_argument('headless')

url = 'https://10.11.227.21/tmui/'
driver.get(url + "login.jsp")

html_source = driver.page_source
print(html_source)

blocStatus = WebDriverWait(driver,    TIMEOUT).until(EC.presence_of_element_located((By.ID, "username")))
inputElement = driver.find_element_by_id("username")
inputElement.send_keys('actualLogin')
inputElement = driver.find_element_by_id("passwd")
inputElement.send_keys('actualPassword')
inputElement.submit()

display.stop()

xvfb is required to use "pyvortualdisplay"

$ sudo apt-get install -y xvfb 
like image 71
frank Avatar answered Nov 09 '22 09:11

frank


Headless Chrome does not support insecure certificates and hence, websites with insecure certificates does not open living it blank. You need to add capabilities as follow:

from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument("--headless")

capabilities = DesiredCapabilities.CHROME.copy()
capabilities['acceptSslCerts'] = True 
capabilities['acceptInsecureCerts'] = True

driver = webdriver.Chrome(chrome_options = chrome_options,executable_path='your path',desired_capabilities=capabilities)
driver.get("yourWebsite")

This will do the work.

like image 25
Suraj Regmi Avatar answered Nov 09 '22 07:11

Suraj Regmi


I had the same issue and setting the window size in conftest.py solved it. My code snippet:

@pytest.fixture
def chrome_options(chrome_options, pytestconfig):
    if pytestconfig.getoption('headless'):
        chrome_options.add_argument('--headless')
        chrome_options.add_argument("window-size=1920,1080")
    else:
        chrome_options.add_argument("start-maximized");
    return chrome_options
like image 34
lukashino Avatar answered Nov 09 '22 08:11

lukashino


Headless chrome may be faster on same machine than headed, try adding some wait before locating password element.

like image 28
mmichalik Avatar answered Nov 09 '22 07:11

mmichalik