Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Scraping with Meteor.js

Can I scrape with meteor.js? Just discovered cheerio which works excellent combined with request. Can I use these with meteor, or is there something similar?

Do you have an working example?

like image 639
kornfridge Avatar asked Feb 09 '13 11:02

kornfridge


2 Answers

Of course! Its hard to imagine what meteor can't do! First you need something to handle the remote http requests. In your meteor directory in the terminal run meteor add http to add the Meteor.http package, also npm install cheerio (have a look at another SO question on how to install npm modules to see exactly where to install external npm modules.

Here is an example that might help you out a bit, it scrapes the current time.

Server js

require = __meteor_bootstrap__.require; //to use npm require must be exposed.
var cheerio = require('cheerio');

Meteor.methods({
    getTime: function () {
        result = Meteor.http.get("http://www.timeanddate.com/worldclock/city.html?n=136");
        $ = cheerio.load(result.content);
        CurrentTime = $('#ct').html();
        return CurrentTime;
    }
});

Client side script:

Meteor.call("getTime", function(error, result) {
    alert("The current time is " + result); 
});

I hope this is helpful. amongst with Cheerio there are also other node frameworks such as node.io

like image 63
Tarang Avatar answered Oct 19 '22 16:10

Tarang


The following code is used in this project to scrape a tweetstorm:

if (Meteor.isClient) {

  Meteor.call('getTweets', function (error, result) {
    if (error) {
      console.log("error", error);
    };

    Session.set("tweets", result);
  });

  Template.tweets.helpers({
    rant: function () {
      return Session.get("tweets");
    }
  });

}

Server side

  if (Meteor.isServer) {
      Meteor.startup(function () {
        var cheerio = Meteor.npmRequire('cheerio');

    Meteor.methods({
      getTweets: function () {
        result = Meteor.http.get("https://twitter.com/Royal_Arse/status/538330380273979393");
        $ = cheerio.load(result.content);
        var body = $('#stream-items-id > li:nth-child(n) > div > div > p').text();
        return body;
      },

    })

  });
}
like image 30
DeBraid Avatar answered Oct 19 '22 15:10

DeBraid