Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Selenium with chromedriver doesn't start via cron

Python script with Selenium and Chromedriver in headless mode on CentOS7 runs fine when called manually.

options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('no-sandbox')
self.driver = webdriver.Chrome(chrome_options=options)

When starting script with crontab however it throws this exception at line 4 (above). Full traceback at bottom.

selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally (Driver info: chromedriver=2.38.552522

Cron is setup with crontab -e

* * * * * cd /to/path && /to/path/.virtualenvs/selenium/bin/python /to/path/script.py -t arg1 arg2 > /to/path/log.txt 2>&1

This produced errors like chromedriver couldn't be found. I then added following to crontab -e.
1) Use bash instead of sh, although starting python script manually from sh works fine
2) Specify path to chromedriver

SHELL=/bin/bash
PATH=/usr/local/bin/

I tried different suggestions found on the web like adding --no-sandbox options to chromedriver in my script. All didn't help. Please note that I am using chrome in headless mode, so I think I don't need this export DISPLAY=:0 stuff in cron, or Xvfb libs as it used to be.

Python 3.6.1
Selenium 3.4.3
Chromedriver 2.38.552522
google-chrome-stable 65.0.3325.181

Full traceback

Exception in thread <name>:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/path/to/script.py", line 53, in start
    self.site_scrape(test_run)
  File "/path/to/script.py", line 65, in site
    self.driver = webdriver.Chrome(chrome_options=options)
  File "/home/<user>/.virtualenvs/selenium/lib64/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
    desired_capabilities=desired_capabilities)
  File "/home/<user>/.virtualenvs/selenium/lib64/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 98, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/home/<user>/.virtualenvs/selenium/lib64/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 188, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/<user>/.virtualenvs/selenium/lib64/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
    self.error_handler.check_response(response)
  File "/home/<user>/.virtualenvs/selenium/lib64/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Linux 4.14.12-x86_64-linode92 x86_64)
like image 882
Jim B Avatar asked May 01 '18 13:05

Jim B


People also ask

How do I run ChromeDriver in headless mode?

ChromeOptions options = new ChromeOptions() options. addArgument("headless"); ChromeDriver driver = new ChromeDriver(options); In the above code, the browser is instructed to run in the headless mode using the addArgument() method of the ChromeOptions class provided by the Selenium WebDriver.

How do I instantiate ChromeDriver?

In order to instantiate the object of ChromeDriver, you can simply create the object with the help of below command. Webdriver driver = New ChromeDriver(); The main motto of the ChromeDriver is to launch Google Chrome. Without that, it is not possible to execute Selenium test scripts in Google Chrome browser.

Does ChromeDriver need to be in path?

By saving chromedriver.exe in the same folder als your Python working directory, there's no need to specify the path.


1 Answers

Finally found the solution. Boy this was bugging me for far too long. Issue was following missing PATH directories: /usr/bin, /usr/sbin in cron. Complete cron looks now like this:

SHELL=/bin/bash
PATH=/usr/local/bin/:/usr/bin:/usr/sbin
* * * * * cd /to/path && /to/path/.virtualenvs/selenium/bin/python /to/path/script.py -t arg1 arg2 > /to/path/log.txt 2>&1
like image 103
Jim B Avatar answered Oct 19 '22 03:10

Jim B