How to run nightmare.js on google appengine for node.js

Question

There is famous issue with nightmare and electron not running on headless linux servers. The official electron docs suggest to use xvfb in order to fake the display. They suggest to use this .yml file for travis.

addons:
  apt:
    packages:
      - xvfb

install:
  - export DISPLAY=':99.0'
  - Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &

Question

How can I use the above piece of code into app.yaml file of google appengine for node.js. I have tried to use this as it is, but glcoud throws an error that addon is invalid command. The gap official docs don't have any similar command.

Any suggestions how we can run nightmare and electron on google appengine for node.js..?

Answer 1

There are two parts related to this question:

1. Running chromium (what electron and, in turn, nightmare "uses") headlessly on linux.
2. Install/Use xvfb to run chromium on app engine.

Part 1)

You need xvfb.

Xvfb (Virtual Framebuffer) is just a program that, from wiki: "is a display server implementing the X11 display server protocol. In contrast to other display servers, Xvfb performs all graphical operations in memory without showing any screen output."

Which is what you need to run a browser without a screen output.

First, install all the xvfb related packages to run it on linux.

apt-get install -y \ xvfb \ x11-xkb-utils \ xfonts-100dpi \ xfonts-75dpi \ xfonts-scalable \ xfonts-cyrillic \ x11-apps \ clang \ libdbus-1-dev \ libgtk2.0-dev \ libnotify-dev \ libgnome-keyring-dev \ libgconf2-dev \ libasound2-dev \ libcap-dev \ libcups2-dev \ libxtst-dev \ libxss1 \ libnss3-dev \ gcc-multilib \ g++-multilib

So with xvfb installed you need to create a virtual xvfb screen and export an environment variable called DISPLAY that points to it. Chromium in Electron will automatically look for $DISPLAY.

The above can be done more easily. Here are two options:

• Calling the program with linux cli (ignore the xvfb warnings if the nightmare script runs fine):

  • xvfb-run -a node main.js. Or...

  • If using rendering related functionality like taking screenshots: xvfb-run -a --server-args="-screen 0 1280x1028x24 -ac +extension GLX +extension RANDR +render" node app.js. Google the xvfb options to adjust to your taste.

• Programmatically: using xvfb npm package

From this point on you should be able to run nightmare on linux.

Part 2)

Nodejs on app engine is ran via the flexible environment. Meaning, through docker containers.

From GAE nodejs runtime: "If your application requires additional operating-system-level dependencies, you will need to use a custom runtime based on this runtime to install the appropriate packages."

Docker is a whole separate topic, but in order to do the above with app engine you have two options as far as I know:

1. Extending the runtime

2. Use GAE with a custom runtime from scratch.

Either way, basically what you would need to do is install the xvfb related packages defining them in the dockerfile and that should do the trick.

Good luck!

Important Notes:

1. The above apt-get packages depend on the availability regarding the linux distro (the above code works on ubuntu and debian). For example, with the specified set of packages and at the time of this post, it will work with GAE's flexible environment since it is based on debian jessie and won't work on linux alpine.

2. Chromium needs a minimum dev/shm allocation to run well. For example, on heroku it is fixed to 5mb -and there is no way to change it. Chromium will crash after a few nightmare actions. So chromium won't work on any heroku's dynos of any size. In docker it is set to 64mb, so depending on the complexity of your script you will do fine or need to adjust it. In plain linux installations, dev/shm is normally set to half of the total available memory. So in a 512mb environment, dev/shm will be set to 256mb and nightmare will most likely run fine.