MongoDB GridFS "illegal chunk format' exception

Question

I've been writing an app in Node.js that stores images in MongoDB's GridFS file system.

I've uploaded images through the app, and the images appear to be stored correctly:

$ mongofiles -v -d speaker-karaoke get howard-basement-100x115.jpg
Tue Jul 17 12:14:16 creating new connection to:127.0.0.1
Tue Jul 17 12:14:16 BackgroundJob starting: ConnectBG
Tue Jul 17 12:14:16 connected connection!
connected to: 127.0.0.1
done write to: howard-basement-100x115.jpg

This grabbed the .jpg out of MongoDB and I was able to open it no problem, so it looks like what I'm uploading is being stored correctly.

However, in my running application, when I attempt to read the same file, I get:

12:15:44 web.1     | started with pid 89621
12:15:45 web.1     | Connecting to mongodb://localhost/speaker-karaoke
12:15:45 web.1     | Speaker Karaoke express app started on 5000
12:15:48 web.1     | DEBUG: Get review thumbnail for 5005b7550333650000000001
12:15:48 web.1     | 
12:15:48 web.1     | node.js:201
12:15:48 web.1     |         throw e; // process.nextTick error, or 'error' event on first tick
12:15:48 web.1     |               ^
12:15:48 web.1     | Error: Illegal chunk format
12:15:48 web.1     |     at Error (unknown source)
12:15:48 web.1     |     at new <anonymous> (/Users/hlship/workspaces/github/speaker-karaoke/node_modules/mongodb/lib/mongodb/gridfs/chunk.js:43:11)
12:15:48 web.1     |     at /Users/hlship/workspaces/github/speaker-karaoke/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:488:24
12:15:48 web.1     |     at Cursor.nextObject (/Users/hlship/workspaces/github/speaker-karaoke/node_modules/mongoose/node_modules/mongodb/lib/mongodb/cursor.js:462:5)
12:15:48 web.1     |     at [object Object].<anonymous> (/Users/hlship/workspaces/github/speaker-karaoke/node_modules/mongoose/node_modules/mongodb/lib/mongodb/cursor.js:456:12)
12:15:48 web.1     |     at [object Object].g (events.js:156:14)
12:15:48 web.1     |     at [object Object].emit (events.js:88:20)
12:15:48 web.1     |     at Db._callHandler (/Users/hlship/workspaces/github/speaker-karaoke/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1290:25)
12:15:48 web.1     |     at /Users/hlship/workspaces/github/speaker-karaoke/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:329:30
12:15:48 web.1     |     at [object Object].parseBody (/Users/hlship/workspaces/github/speaker-karaoke/node_modules/mongoose/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:118:5)
12:15:48 web.1     | process terminated
12:15:48 system    | sending SIGTERM to all processes

Using this code (CoffeeScript, for Express):

  app.get "/images/review-thumbnail/:id", (req, res) ->

    id = req.params.id

    util.debug "Get review thumbnail for #{id}"

    store = new GridStore mongoose.connection.db, new ObjectID(id), null, "r"

    store.open (err, file) ->

      throw err if err

      util.debug "Store open for #{id}, type = #{file.contentType}"

      # TODO: Scale the image before sending it!

      res.header "Content-Type", file.contentType

      store.stream(true).pipe res

So it doesn't seem to be even making it to the callback passed to store.open().

Is there a problem opening a GridFS file when you know the id, but not the file name?

BTW:

$ npm ls
[email protected] /Users/hlship/workspaces/github/speaker-karaoke
├─┬ [email protected]  extraneous
│ ├── [email protected] 
│ ├── [email protected] 
│ └── [email protected] 
├── [email protected] 
├─┬ [email protected] 
│ ├── [email protected] 
│ ├── [email protected] 
│ ├─┬ [email protected] 
│ │ ├── [email protected] 
│ │ ├── [email protected] 
│ │ └── [email protected] 
│ └── [email protected] 
├─┬ [email protected] 
│ ├─┬ [email protected] 
│ │ └── [email protected] 
│ ├── [email protected] 
│ ├── [email protected] 
│ └── [email protected] 
├─┬ [email protected] 
│ ├── [email protected] 
│ └── [email protected] 
├─┬ [email protected] 
│ └── [email protected] 
├─┬ [email protected] 
│ ├── [email protected] 
│ └─┬ [email protected] 
│   └── [email protected] 
├─┬ [email protected] 
│ └── [email protected] 
├─┬ [email protected] 
│ ├─┬ [email protected] 
│ │ ├── [email protected] 
│ │ └── [email protected] 
│ └── [email protected] 
├── [email protected] 
└── [email protected] 

And here's the function where it is failing:

var Chunk = exports.Chunk = function(file, mongoObject) {
  if(!(this instanceof Chunk)) return new Chunk(file, mongoObject);

  this.file = file;
  var self = this;
  var mongoObjectFinal = mongoObject == null ? {} : mongoObject;

  this.objectId = mongoObjectFinal._id == null ? new ObjectID() : mongoObjectFinal._id;
  this.chunkNumber = mongoObjectFinal.n == null ? 0 : mongoObjectFinal.n;
  this.data = new Binary();

  if(mongoObjectFinal.data == null) {
  } else if(typeof mongoObjectFinal.data == "string") {
    var buffer = new Buffer(mongoObjectFinal.data.length);
    buffer.write(mongoObjectFinal.data, 'binary', 0);
    this.data = new Binary(buffer);
  } else if(Array.isArray(mongoObjectFinal.data)) {
    var buffer = new Buffer(mongoObjectFinal.data.length);
    buffer.write(mongoObjectFinal.data.join(''), 'binary', 0);
    this.data = new Binary(buffer);
  } else if(mongoObjectFinal.data instanceof Binary || Object.prototype.toString.call(mongoObjectFinal.data) == "[object Binary]") {    
    this.data = mongoObjectFinal.data;
  } else if(Buffer.isBuffer(mongoObjectFinal.data)) {
  } else {
    throw Error("Illegal chunk format");
  }
  // Update position
  this.internalPosition = 0;
};

Solution

Updating the solution here as it does not render correctly in the comments below.

The problem was the duplication; having two copies, even with the same version, of mongodb and bson.

Fortunately, mongoose exports the mongodb it requires as property mongo, so I was able to remove the explicit mongodb from my package.json and changed:

mongo = require "mongodb"
mongoose = require "mongoose"

to:

mongoose = require "mongoose"
mongo = mongoose.mongo

Things are now looking nice; I still think the module system needs a sanctioned way to access a dependencies dependencies (for the case where a dep is not thoughtful enough to expose its deps).

Answer 1

Pretty sure it was duplicate copies (same version) of mongodb module; removing node_modules and nmp install seems to have fixed it.