node.js fs - stream file "backwards" - from bottom to top

Question

Using Node.js, what is the best way to stream a file from a filesystem into Node.js, but reading it backwards, from bottom to top? I have a large file, and there doesn't seem to be much sense in reading from the top if I only want the last 10 lines. Is this possible?

Right now I have this horrible code, where we do a GET request with a browser to view the server logs, and pass a query string parameter to tell the server how many lines at the end of the log file we want to read:

function get(req, res, next) {

    var numOfLinesToRespondWith = req.query.num_lines || 10;

    var fileStream = fs.createReadStream(stderr_path, {encoding: 'utf8'});

    var jsonData = [];   //where jsonData gets populated
    var ret = [];

    fileStream.on('data', function processLineOfFileData(chunk) {
        jsonData.push(String(chunk));
    })
    .on('end', function handleEndOfFileData(err) {
        if (err) {
            log.error(colors.bgRed(err));
            res.status(500).send({"error reading from smartconnect_stdout_log": err.toString()});
        }
        else {

         for(var i = 0; i < numOfLinesToRespondWith; i++){
                ret.push(jsonData.pop());
           }

          res.status(200).send({"smartconnect_stdout_log": ret});

        }
    });
}

the code above reads the whole file and then adds the number of lines requested to the response after reading the whole file. This is bad, is there a better way to do this? Any recommendations will be met gladly.

(one problem with the code above is that it's writing out the last lines of the log but the lines are in reverse order...)

One potential way to do this is:

process.exec('tail -r ' + file_path).pipe(process.stdout);

but that syntax is incorrect - so my question there would be - how do I pipe the result of that command into an array in Node.js and eventually into a JSON HTTP response?

Answer 1

I created a module called fs-backwards-stream that could may meet your needs. https://www.npmjs.com/package/fs-backwards-stream

If you need the result parsed by lines rather than byte chunks you should use the module fs-reverse https://www.npmjs.com/package/fs-reverse or

both of these modules stream you could simply read the last n bytes of a file.

here is an example using plain node fs apis and no dependencies.

https://gist.github.com/soldair/f250fb497ce592c3694a

hope that helps.