How to read a file by setting correct offset and position and write to the response in Nodejs with manual buffering?

Question

I want to read a file in 64byte interval. And I also do not want to use any functionality which interanlly implements buffering. I wanted to do buffering manually. So I started using fs.read(). I tried hard but I really don't know how to set position which tells where to read from in the file and offset in the buffer to start writing at.
So I found few resources and started implementing by my own. But what I did seems enterly wrong. Please find my code below.

app.get('/manualBufferAnother', function (req, res, next) {
   var filePath = path.join(__dirname, 'Koala.jpg');
   console.log("FilePath is: "+filePath);
   var fileName = path.basename(filePath);
   var mimeType = mime.lookup(filePath);
   var stat = fs.statSync(filePath);

   res.writeHead(200, {
         "Content-Type": mimeType,
         "Content-Disposition" : "attachment; filename=" + fileName,
         'connection': 'keep-alive',
         "Content-Length": stat.size,
         "Transfer-Encoding": "chunked"
   });

   fs.open(filePath, 'r', function(err, fd) {
       var completeBufferSize = stat.size;
       var offset = 0;  //is the offset in the buffer to start writing at
       var length = 511; //is an integer specifying the number of bytes to read
       var position = 0;  //is an integer specifying where to begin reading 
       from in the file. If position is null, data will be read from the current file position
       var buffer = new Buffer(completeBufferSize);
       buf(res,fd,offset,position,length,buffer,stat);        
   });
 });

var buf = function(res,fd,offset,position,length,buffer,stat) {
if(position+buffer.length < length) {
    fs.read(fd,buffer,offset,length,position,function(error,bytesRead,bufferr {
        res.write(bufferr.slice(0,bytesRead));
        console.log("Bytes Read: "+bytesRead);
        position=position+bufferr.length;
        buf(res,fd,offset,position,length,bufferr,stat);
    })
} else {
    fs.read(fd,buffer,offset,length,position,function(error,bytesRead,bufferr) {
        console.log("Bytes Read in else: "+bytesRead);
        res.end(bufferr.slice(0,bytesRead));
        fs.close(fd)
    })
}
}

I know this code is doing so much wrong thing. But I don't know the right way. Should I use any loop for setting and storing position and offset values? Will be really helpful if you provide me good reference?

Answer 1

Here is an example:

res.writeHead(...);
var SIZE = 64; // 64 byte intervals
fs.open(filepath, 'r', function(err, fd) {
  fs.fstat(fd, function(err, stats) {
    var bufferSize = stats.size;
    var buffer = new Buffer(bufferSize),
    var bytesRead = 0;

    while (bytesRead < bufferSize) {
      var size = Math.min(SIZE, bufferSize - bytesRead);
      var read = fs.readSync(fd, buffer, bytesRead, size, bytesRead);
      bytesRead += read;
    }
    res.write(buffer);
  });
});

Should I use any loop for setting and storing position and offset values?

Yes you can but be careful. In Node.js, most file system functions are asynchronous (non-blocking). As you probably realised, putting an asynchronous function in a loop is going to cause problems. You can tell if a function is asynchonous by looking at the Node.js documentation and checking if there is a callback parameter. So using read inside a loop is bad. We can instead use readSync. This is the synchronous (blocking) and is similar to C's read() function (which is also blocking).

I really don't know how to set position which tells where to read from the file and offset in the buffer to start writing at.

The arguments of the readSync function control both where to read from in the file and where in the destination buffer to write to.

//                        /----- where to start writing at in `buffer`    
fs.readSync(fd, buffer, offset, length, position)
//                                           \------- where to read from in the 
//                                                    file given by `fd`

---

Note: The above style is idiomatic for C, but in Javascript it is considered poor practice -- the code will not scale well. In general you don't want to use synchronous functions ever because they block the single thread of execution that Javascript uses (aka "blocking the event loop").
From Express.js:

Synchronous functions and methods tie up the executing process until they return. A single call to a synchronous function might return in a few microseconds or milliseconds, however in high-traffic websites, these calls add up and reduce the performance of the app. Avoid their use in production. Although Node and many modules provide synchronous and asynchronous versions of their functions, always use the asynchronous version in production.

Using .pipe() and Streams (asynchronous style) is generally the way to go if you want the most idiomatic and performant code. Sorry to say this: no official sources / popular websites will describe file operations using synchronous functions via C-style blocking and buffering because it is bad practice in Node.js.