Flask, Nginx, Gunicorn Stack Launching Selenium instance

Question

I have a user who I have added to the www-data group. I have a service setup to run an API that launches a Selenium instance. I am able to launch the code as my user, but the web app fails to launch. I originally had problems with Xvfb but after adding the path to the environment of the service file it was resolved. I am now having a similar issue with google-chrome not executing.

I've tried adding path to environment, and changing permissions on files and folders.

nginx conf

server {
        listen 443 ssl;

        root /usr/share/nginx/html;
        index index.html index.htm;

        server_name private;
        location / {
               proxy_pass http://127.0.0.1:8888;
               proxy_set_header Host $host;
               proxy_set_header X-Real-IP $remote_addr;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

    ssl_certificate /etc/letsencrypt/live/private/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/private/privkey.pem; # managed by Certbot
}
server {
    if ($host = private) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80 default_server;
        listen [::]:80 default_server;

        server_name private;
    return 404; # managed by Certbot
}

[Unit]
Description=Gunicorn serving hrapi
After=network.target

[Service]
User=artem
Group=www-data
WorkingDirectory=/home/artem/ucs
Environment="PATH=/home/artem/env/bin:/usr/bin"
ExecStart=/home/artem/env/bin/gunicorn --workers 3 --bind 127.0.0.1:8888 wsgi:app

[Install]
WantedBy=multi-user.target

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"fname": "ijay", "lname": "private", "addr": "private", "city": "scottsdale", "state":"AZ", "postal":"private", "ssn":"private"}' https://hrapi.domain.com/getreport
Error:b'Traceback (most recent call last):
  File "/home/artem/ucs/get_report.py", line 158, in <module>
    b = UcsBot(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5], sys.argv[6], sys.argv[7])
  File "/home/artem/ucs/get_report.py", line 40, in __init__
    self.driver = webdriver.Chrome(\'/home/artem/ucs/chromedriver\', chrome_options=options)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn\'t exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Selenium is supposed to launch, and it does launch normally when I run the script. The script can't be launched from the API though.

Also, I have ensured that my chrome and chromedriver are correct versions. The script executes fine by itself from user as I said before just not when triggered via the API.

From the verbose log so it seems the problem is still permissions as suspected. j

[1563464973.354][INFO]: Launching chrome: /usr/bin/google-chrome --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --disable-web-resources --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --force-fieldtrials=SiteIsolationExtensions/Control --headless --ignore-certificate-errors --load-extension=/tmp/.com.google.Chrome.rpiwaU/internal --log-level=0 --no-first-run --no-sandbox --password-store=basic --remote-debugging-port=0 --test-type=webdriver --use-mock-keychain --user-data-dir=/tmp/.com.google.Chrome.aQmopJ data:,
/usr/bin/google-chrome: line 8: readlink: command not found
/usr/bin/google-chrome: line 24: mkdir: command not found
/usr/bin/google-chrome: line 45: exec: cat: not found
/usr/bin/google-chrome: line 46: exec: cat: not found
[1563464973.406][INFO]: [60f2bc8b762ce68091b9abc2d8476bf5] RESPONSE InitSession ERROR unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
[1563464973.406][DEBUG]: Log type 'driver' lost 0 entries on destruction
[1563464973.406][DEBUG]: Log type 'browser' lost 0 entries on destruction 

After this I did

sudo chown artem:www-data /usr/bin/google-chrome
sudo chown artem:www-data /usr/bin/chromedriver

Still same problem.

# wsgi.py

from api import app
if __name__=="__main__":
    app.run()

Answer 1

Add :/bin to your PATH as shown below:

Environment="PATH=/home/artem/env/bin:/usr/bin:/bin"

mkdir, readlink etc are in /bin path which is not in the modified PATH

Answer 2

The chrome startup script /usr/bin/google-chrome has a problem, as indicated by the /usr/bin/google-chrome: line 24: XXX: command not found:

[1563464973.354][INFO]: Launching chrome: /usr/bin/google-chrome [...]
/usr/bin/google-chrome: line 8: readlink: command not found
/usr/bin/google-chrome: line 24: mkdir: command not found
/usr/bin/google-chrome: line 45: exec: cat: not found
/usr/bin/google-chrome: line 46: exec: cat: not found

It might be a simple $PATH problem, meaning that the uWSGI server does not find all the mentioned commands readlink etc. because of it does not have these commands within the $PATH variable.

Warning: I would not allow www-data to access all the command XXX by changing their permissions, these would be a major security issue on a production server!

Instead, I suggest to look at the following SO question: How to fix "usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed" error in Linux?

Alternative: Can you somehow solve your job without using Chrome? What are your reasons for starting an Chrome as a service on a webserver?