Record and process screen capture at high frame rate in NodeJS

Question

As stated in the title, I would like to capture the screen 60 times a second and run it through a nodejs function.

Most libraries on npm only save to disk.

I tried taking screenshots with robotjs but only achieved 11fps.

I tried starting an RTMP server and using OBS but its delayed.

What in the world do I do?

Answer 1

You can use ffmpeg to achieve 60fps. Take a look at ffmpeg docs for more details.

const ffmpeg = require("ffmpeg-static").path;
const { spawn } = require("child_process");

const process = spawn(
    ffmpeg,
  ["-probesize", "10M", "-f", "gdigrab", "-framerate", "60", "-i", "desktop", "-f", "flv", "-"],
  { stdio: "pipe" }
);
const stream = process.stdout;

Here you have the stream of video. Do the processing.

For example, you can write the stream to a Writable stream

const { createWriteStream } = require("fs");

const file = createWriteStream("capture.flv");
stream.pipe(file);

Or you can subscribe to chunks of data and do some processing

stream.on("data", chunk => {
  console.log(chunk);
});

More about Node.JS streams

Note:

This example would only work for Windows users. Mac and Linux users should change ffmpeg options used to spawn the process as described in the ffmpeg docs.

Answer 2

Your best bet might be to create a native process for doing the capture, using platform-specific APIs, then wrap them in NAN (Native Abstraction for Node: https://github.com/nodejs/nan). There's a little learning curve, but not too bad. I've built systems for decoding JPEGs, manipulating them, and encoding in h.264 this way.

Because you're in control of the capture at the native level, you can create whatever interface you like to bring screen grabs into Node. For example, you might want to avoid marshaling images across the native-javascript interface and instead maintain handles to them, and only pull an image when you need to.

Then you can turn it all into a cool npm module and be adored by everyone.