Stdout buffer issue using node child_process

Question

I'm trying to execute curl using node child_process to get a JSON file (about 220Ko) from a shared folder in a local network. But it actually returns a buffer problem that I can't get throught. Here is my code:

var exec = require('child_process').exec;

var execute = function(command, callback){
    exec(command, function(error, stdout, stderr){ callback(error, stdout); });
};

execute("curl http://" + ip + "/file.json", function(err, json, outerr) {
    if(err) throw err;
    console.log(json);
})

And here is the error I get:

if(err) throw err;
          ^
Error: stdout maxBuffer exceeded.
    at Socket.<anonymous> (child_process.js:678:13)
    at Socket.EventEmitter.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:746:14)
    at Socket.EventEmitter.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:408:10)
    at emitReadable (_stream_readable.js:404:5)
    at readableAddChunk (_stream_readable.js:165:9)
    at Socket.Readable.push (_stream_readable.js:127:10)
    at Pipe.onread (net.js:526:21)

Answer 1

You need to use and set the maxBuffer option when using child_process.exec. From the documentation:

maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed.

The documentation also states that the default value of maxBuffer is 200KB.

As an example, the maximum buffer size is increased to 500KB in the following code:

var execute = function(command, callback){
    exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ callback(error, stdout); });
};

Additionally, you may want to read about http.get to see if it is capable of achieving what you are trying to do.

Answer 2

I had a similar issue and I fixed it moving from exec to spawn:

var child = process.spawn('<process>', [<arg1>, <arg2>]);

child.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

child.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

child.on('close', function (code) {
    console.log('child process exited with code ' + code);
});

Answer 3

Adding some explanation to the answers.

The exec command buffers the data before sending it to the parent process. It is generally suitable for the commands producing the smaller output. The above error occurred because the output generated by the execution of the command was larger than the max buffer size. One way to solve the above issue is to specify the buffer size as answered by Tim Cooper.

var execute = function(command, callback){
exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ 
 callback(error, stdout); });
};

Another solution is to use the spawn method which is generally faster compared to the exec and it does not buffer the data before sending. It sends the data as a stream hence the problem of buffer size never occurs. The code snippet used by Isampaio.

var child = process.spawn('<process>', [<arg1>, <arg2>]);
child.stdout.on('data', function (data) {
 console.log('stdout: ' + data);
});
child.stderr.on('data', function (data) {
 console.log('stderr: ' + data);
});
child.on('close', function (code) {
 console.log('child process exited with code ' + code);
});