AWS Lambda Node.js 10.x Runtime error with selenium-webdriver

Question

A few days back we received a notification regarding 'Lambda operational notification' to update our Node.js 8.10 runtime to Node.js 10.x runtime.

In response to this notification, we installed Node.js version v10.16.3 in our development system and tested our existing code. We found the code was running fine in our development system, but when we tested this same code in AWS Lambda with Node.js 10.x runtime we get this following error:

2019-10-28T12:03:31.771Z 8e2472b4-a838-4ede-bc70-a53aa41d9b79 INFO Error: Server terminated early with status 127 at earlyTermination.catch.e (/var/task/node_modules/selenium-webdriver/remote/index.js:251:52) at process._tickCallback (internal/process/next_tick.js:68:7)

'aws-sdk', 'selenium-webdriver' npm packages and google chrome binaries are the only dependencies used in our project.

Our project has the following file structure.

/var/task/
├── index.js
├── lib
│   ├── chrome
│   ├── chromedriver
│   ├── libgconf-2.so.4
│   ├── libORBit-2.so.0
│   └── libosmesa.so
└── node_modules
    ├── selenium-webdriver
    ├── ...

Since this code is not throwing any error in our development system, we suspect it has to do with the new runtime.

We tried the setting the binary path using setChromeBinaryPath()

This is the code we are using. The error occurs when the build() method is called.

var webdriver = require('selenium-webdriver');
var chrome = require('selenium-webdriver/chrome');
var builder = new webdriver.Builder().forBrowser('chrome');
var chromeOptions = new chrome.Options();
const defaultChromeFlags = [
    '--headless',
    '--disable-gpu',
    '--window-size=1280x1696', // Letter size
    '--no-sandbox',
    '--user-data-dir=/tmp/user-data',
    '--hide-scrollbars',
    '--enable-logging',
    '--log-level=0',
    '--v=99',
    '--single-process',
    '--data-path=/tmp/data-path',
    '--ignore-certificate-errors',
    '--homedir=/tmp',
    '--disk-cache-dir=/tmp/cache-dir'
];

chromeOptions.setChromeBinaryPath("/var/task/lib/chrome");
chromeOptions.addArguments(defaultChromeFlags);
builder.setChromeOptions(chromeOptions);

var driver = await builder.build();

Answer 1

We recently faced the exact same issue. After upgrading to AWS Lambda Node v10.x from Node v8.x, chrome and chromedriver stopped working. In short, the root cause is that Lambda Node 10.x runs on Amazon Linux 2 Vs Lambda Node v8 which runs on Amazon Linux. Amazon Linux 2 lacks a number of packages comparing to it's predecessor, making it more lightweight but at the same time a pain in case you want to set up a custom runtime environment. Before I give you the steps to resolve this, let me first highlight a few useful links that helped me find the right set of binaries I had to also include in my lambda deployment package.

Just remember! The way to resolve this is to figure out which binaries are missing from your Lambda deployment package and add them in.

1. How to use Amazon Linux native binary packages in an AWS Lambda deployment package. Once you know you are missing some binaries in your Lambda environment, this link from AWS will help you include them into your package. For my purpose I used an EC2 Amazon Linux 64 bit AMI to download the packages and extract them. Detailed steps follow... https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package
2. Besides binaries missing from Amazon Linux 2, there are also no fonts installed. This link will tell you how to install fonts on AWS Lamda. One of the reasons Chrome fails to run on Lambda is the lack of fonts. https://forums.aws.amazon.com/thread.jspa?messageID=776307
3. This is a nice issue thread on github that taught me that the order of paths in the LD_LIBRARY_PATH environment variable matters. This is the environment variable that holds the paths where your binaries are in. https://github.com/alixaxel/chrome-aws-lambda/issues/37
4. Now this is a game changer. Without the amazing docker container lambci created simulating AWS Lambda to as close as it can be, I would have never figured it out. After trying all sorts of things between an Amazon Linux 2 EC2 server and AWS Lambda, this ended up being my playground, where I could iterate trying different packages very quickly. https://hub.docker.com/r/lambci/lambda/
5. Running Arbitrary Executables in AWS Lambda. Some helpful link if you want to run an executable directly on lambda and see how it behaves. The error messages you see from selenium-webdriver package are actually not surfacing the real error that chrome or chromedriver throws. Trying to directly run chrome or chromedriver in the lambci docker container is how I managed to debug this and figure out which binaries were missing. https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/

So, here is what you need to do:

1. Launch an Amazon Linux 2 64 bit server. A t3.micro should be enough.
2. SSH to the machine and install rmpdevtools: sudo yum install -y yum-utils rpmdevtools
3. Create a temporary directory for downloading the missing packages:
4. cd /tmp
5. mkdir lib
6. cd lib
7. Download the RPM packages missing in AWS Lambda node v10.x: yumdownloader --resolve GConf2 glibc glib2 libblkid libffi libgcc libmount libsepol libstdc++ libuuid pcre zlib libselinux dbus-glib mozjs17 polkit polkit-pkla-compat libX11 libX11-common libXau libxcb fontconfig expat fontpackages-filesystem freetype stix-fonts gnu-free-sans-fonts fontpackages-filesystem gnu-free-fonts-common nss nspr nss-softokn nss-softokn-freebl nss-util dbus-libs audit-libs bzip2-libs cracklib elfutils-libelf elfutils-libs libattr libcap libcap-ng libcrypt libdb libgcc libgcrypt libgpg-error libsepol lz4 pam systemd-libs xz-libs mesa-libOSMesa-devel mesa-libOSMesa mesa-libglapi sqlite
8. Extract the RPM packages: rpmdev-extract *rpm
9. Create some temporary location for copying the binaries from the extracted RPM artifacts:
   • sudo mkdir -p /var/task
   • sudo chown ec2-user:ec2-user /var/task
   • cd /var/task
   • mkdir lib
   • mkdir fonts
10. Copy the extracted binaries to the new temporary location:
    • /bin/cp /tmp/lib/*/usr/lib64/* /var/task/lib
    • /bin/cp /tmp/lib/*/lib64/* /var/task/lib
    • /bin/cp /tmp/lib/*/usr/share/fonts/*/*.ttf /var/task/fonts
11. Zip the artifacts: zip -r ./lib.zip ./*
12. Download them from the server extract the zip and include your lambda handler. At this point you should have a very similar structure like the one you had with some more binaries in your lib folder and a new fonts folder.
13. Include the following config file "fonts.conf" in your /var/task/fonts folder:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <dir>/var/task/fonts/</dir>
  <cachedir>/tmp/fonts-cache/</cachedir>
  <config></config>
</fontconfig>

13. Add the following code snippet in your lambda handler. This will set the right order of include paths for the LD_LIBRARY_PATH environment variable and will also set the FONTCONFIG_PATH to the new /var/task/fonts directory.

process.env.FONTCONFIG_PATH = `${process.env.LAMBDA_TASK_ROOT}/fonts`;
if (process.env.LD_LIBRARY_PATH.startsWith("/var/task/lib:") !== true) {
  process.env.LD_LIBRARY_PATH = [...new Set(["/var/task/lib", ...process.env.LD_LIBRARY_PATH.split(':')])].join(':');
}

14. Download locally lambci/lambda image. docker pull lambci/lambda
15. Debug your lamda handler by running a lambci image like this:

docker run --rm -v "$THE_LOCAL_DIR_OF_YOUR_UNCOMPRESSED_LAMDA_PACKAGE":/var/task lambci/lambda:nodejs10.x index.handler

16. Iterate steps 7 to 14 until you get it working on the lambci container. With the given RPM packages it should work, but in case it does not, you can debug locally what is going on by trying to launch chrome in your lambda like this:

const childProcess = require('child_process');
childProcess.execFileSync(`${process.env.LAMBDA_TASK_ROOT}/lib/chrome`);

This is a cumbersome process, but at the end of the day, all you are doing is just adding some more binaries into your package and 3 lines of code in your handler to update the lib and fonts environment variables.

Just in case, adding below as well the chrome flags we are using:

const defaultChromeFlags = [
  "--headless",
  "--disable-gpu",
  "--window-size=1280x1024",
  "--no-sandbox",
  "--user-data-dir=/tmp/user-data",
  "--hide-scrollbars",
  "--enable-logging",
  "--v=99",
  "--single-process",
  "--data-path=/tmp/data-path",
  "--ignore-certificate-errors",
  "--homedir=/tmp",
  "--disk-cache-dir=/tmp/cache-dir"
];

Good luck!