Using multer with inmemory option and streaming into mongodb gridfs

Question

The following priece of code works as expected: it reads a file that has been uploaded using multer [https://github.com/expressjs/multer] and streams it into gridfs. Im also able to steam this out of gridfs as well.

var target = gfs.createWriteStream({
            filename: fileItem.originalname,
            mode: 'w',
            metadata: metaData
        });

        fs.createReadStream(tempFile)
            .on('end', function () {
                console.log('File read and stored');
            })
            .on('error', function (err) {
                console.log('Error handling file ' + err);
            })
            .pipe(target);

        // Attach new file to document list
        target.on('close', function (file) {
            filesUploadedList.push(file._id);

fileItem is obtained from iterating over req.files.file which are the files uploaded.

However, I'm trying to figure out the inMemory option for multer. If I set this to true then the fileItem will have the buffer populated. Just to note without this option the buffer/contents of the fileItem is empty which is why in the code above its reading from the fileItem.location.

What is the best way to populate the target with contents? the fs.createReadStream is doing the piping at the moment.

Thanks.

J

Answer 1

You can accomplish this by using streamifier. The code looks should look like this:

streamifier.createReadStream(fileItem.buffer).pipe(target);

Multer does not write anything to disk, target is now populated with a the buffer/contents of the file being uploaded.

Answer 2

You could use it this way :

(original idea here: stored files directly in mongodb with gridfs-stream)

app.post('/upload',multer({
    dest:"./tmp/",
    upload:null,// take uploading process 
    inMemory:true, //or false, not needed here

    onFileUploadStart:function(file){
       //set upload with WritableStream        
       this.upload = gfs.createWriteStream({
           filename:file.originalname,
           mode:"w",
           chunkSize:1024*4,
           content_type:file.mimetype,
           root:"fs",
           metadata: {} // put some crazy meta data in here
       });
    },

    onFileUploadData:function(file,data) {
       //put the chunks into db 
       this.upload.write(data);
    },

    onFileUploadComplete:function(file) {
       //end process 
       this.upload.end();
       console.log('successfully written File to MongoDB Gridfs');
    }
    }),
    function(req,res) {
       res.sendStatus(200);
   });

Since you are using the onFileUploadData()-function the Gridsfs would be populated as data arrives. This works regardlessly of setting inMemory to true or false. You should note, that the file.buffer[] can get very large if you are dealing with huge files. It's just a copy of all incoming data fragments.

In theory it might be better to use an own defined middleware on top of busboy without performing file operations (as used in Multer) at all.