Stream stdout from child process to browser via expressjs

Question

We have an app built using nodejs, express and child_process.spawn. One requirement is that we need to spawn a process at runtime and capture it's output and present it to the user.

We have that working. However, we need to figure out a way to stream the output instead of waiting until the child process exists.

We've looked around and couldn't find any clear cut examples and were wondering if anyone here had any ideas?

Everything is working. We just don't care for the user experience of having to wait for the command to finish before showing the entire output. If we could stream it, that would be ideal so as the stdout data event was triggered, the browser would update with the new data. As right now it does come in chunks as opposed to one big blob.. so it's suited quite nicely to do this.

Answer 1

The response object of an Express route is also an instance of writable stream, so that allows you to pipe the child process' stdio streams to the response.

app.get('/path', function(req, res) {
  var child = spawn('ls', ['-al']);
  child.stdout.pipe(res);
});

Answer 2

Another approach would be to use Socket.IO. Socket.IO is designed for this kind of real-time event-based communication between the client and the server.

The key is to flush the standard output stream after writing to it. This should trigger the "childprocess.stdout.on('data', ...)" event.

How this is done depends on what language your child process is written in. Say you're running a python script and writing to the standard output using "print", then you'd need to do this:

import sys

# Flush after each print to stdout
sys.stdout.flush()

Here's a simple example that implements what you're asking using Socket.IO. Server side code:

var app = require('express')();
var http = require('http').Server(app);
var io = require('socket.io')(http);
var child = require('child_process');

app.get('/', function(req, res){
  res.sendFile(__dirname + '/index.html');
});

http.listen(8000, function(){
	console.log('listening on *:8000');
});

io.on('connection', function(socket){
	console.log('new user connected');
	
	var python = child.spawn( 'python', ['compute.py'],[]);

	var chunk = '';
	python.stdout.on('data', function(data){
		chunk += data
		socket.emit('newdata', chunk);
	} );
	
	python.stderr.on('data', function (data) {
		console.log('Failed to start child process.');
	})
})

On the client, you could then register an event listner:

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io();
   
  socket.on('newdata', function(d){
     // Do something with the data
  })
</script>