Downloading remote images using Meteor (CFS)

Question

So, I'm trying to figure out how to download a remote image, and then store the downloaded image using CollectionFS.

I was trying to use the automatic URL handling in CFS, but the host I'm downloading images -from- has HEAD requests disabled, so I can't use it.

I was either going to use Meteor.get, or NPM's 'request', but I don't really understand how to combine the two to get the desired result.

Any thoughts would be be greatly appreciated. All I know how to do is use the URL in a Meteor.get request, but after that, I'm really lost.

This is sort of what I get so far, but I don't know what to do with the result of the request afterwards:

var result = HTTP.get(url);

I only assume that I'm supposed to do something with result.body (as per the Meteor documentation), but I don't know how to properly encode that object so that it can be shoved into a CFS collection locally.

Answer 1

From what I've read on the CollectionFS API, server-side inserts can take a Node.js Buffer object as parameter.

https://github.com/CollectionFS/Meteor-CollectionFS#getting-started

A Buffer object is what you'll get from npm request package with an encoding set to null, and this is what you're expected to insert into CollectionFS.

If we don't set encoding to null, the response will pass through string encoding that will break our image data since this is raw binary.

Unfortunately you can't use Meteor HTTP package to do this because it acts as a wrapper around npm request and specifically force the encoding to utf-8 as seen in line 74 :

https://github.com/meteor/meteor/blob/devel/packages/http/httpcall_server.js#L74

As you probably know, npm packages are not directly usable in Meteor because server-side environment relies on Fibers.

So here is the necessary wrapping around request, as a yet unreleased package :

/packages/request/package.js :

Package.describe({
  summary:"Simplified HTTP request client",
  version:"2.40.0"
});

Npm.depends({
  "fibers":"1.0.1",
  "request":"2.40.0"
});

Package.onUse(function(api){
  //
  api.versionsFrom("[email protected]");
  //
  api.use("underscore","server");
  //
  api.addFiles("server/lib/request.js","server");
  //
  api.export("request","server");
});

/packages/request/server/lib/request.js :

var Future=Npm.require("fibers/future");
request=Npm.require("request");

var requestSync=function(uri,options){
  var future=new Future();
  request(uri,options,function(error,response,body){
    if(error){
      console.log(error);
      throw error;
    }
    future.return({
      response:response,
      body:body
    });
  });
  return future.wait();
};

_.extend(request,{
  putSync:function(uri,options){
    options.method="PUT";
    return requestSync(uri,options);
  },
  patchSync:function(uri,options){
    options.method="PATCH";
    return requestSync(uri,options);
  },
  postSync:function(uri,options){
    options.method="POST";
    return requestSync(uri,options);
  },
  headSync:function(uri,options){
    options.method="HEAD";
    return requestSync(uri,options);
  },
  delSync:function(uri,options){
    options.method="DELETE";
    return requestSync(uri,options);
  },
  getSync:requestSync
});

Then you can use request like this :

var result=request.getSync(url,{
  encoding:null
});
var buffer=result.body;

The buffer variable will hold the unaltered image data you need to pass to CollectionFS insert.

Answer 2

At some point Meteor added npmRequestOptions as a way to override their reasonable defaults for cases like this.

You can now simply do

HTTP.get(url, {npmRequestOptions: {encoding: 'binary'}})

to get correct image data