Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Use Nightmare.js without ES6 syntax and yield

I built a simple node script using nightmare.js to scrape websites

var Nightmare = require('nightmare');
var vo = require('vo');

vo(run)(function(err, result) {
    if (err) throw err;
});

function *run() {
    var x = Date.now();
    var nightmare = Nightmare();
    var html = yield nightmare
    .goto('http://google.com')
    .evaluate(function() {
        return document.getElementsByTagName('html')[0].innerHTML;
    });

    console.log("done in " + (Date.now()-x) + "ms");
    console.log("result", html);

    yield nightmare.end();
}

I want to run this in an environment using an older version of node, which does not support ES6 features. There are no examples on the github page on how to do this without the "yield" keyword.

I did find an example of usage without the ES6 syntax here : Webscraping with nightmare

I wrote it like this :

var night = new Nightmare()
.goto('http://www.google.com')
.evaluate(function () {
  return document.getElementsByTagName('html')[0].innerHTML;
},function (html) {
   console.log("result", html);
  }
)
.run(function (err, nightmare) {
  if (err) return console.log(err);
  console.log('Done!');
});

It does not crash, but the result logging function is never called.

With the yield syntax, getting the returned value from "evaluate" is pretty straightforward, but without it, I did not find any way to do it.

UPDATE Wrote this thanks to the accepted answer and its comments. It uses 'Q' and works in node versions previous to 0.12:

var Nightmare = require('nightmare');

var Promise = require('q').Promise;

var x = Date.now();
var nightmare = Nightmare();
Promise.resolve(nightmare
  .goto('http://google.com')
  .evaluate(function() {
      return document.getElementsByTagName('html')[0].innerHTML;
})).then(function(html) {
    console.log("done in " + (Date.now()-x) + "ms");
    console.log("result", html);
    return nightmare.end();
}).then(function(result) {

}, function(err) {
   console.error(err); // notice that `throw`ing in here doesn't work
});
like image 893
Rayjax Avatar asked Sep 15 '15 14:09

Rayjax


1 Answers

The docs are horrible, but it seems that Nightmare is based on thenables. I didn't find much information on the callback interface either, but that would lead to an indentation pyramid anyway.

So your best bet is to use promises, just choose any library that roughly follows the ES6 standard (they all are usable in non-ES6 environments as well).

You can easily transform your linear generator code into a promise chain, just replace every yield by a then call:

var Nightmare = require('nightmare');
var Promise = require('…');

var x = Date.now();
var nightmare = Nightmare();
Promise.resolve(nightmare
  .goto('http://google.com')
  .evaluate(function() {
      return document.getElementsByTagName('html')[0].innerHTML;
})).then(function(html) {
    console.log("done in " + (Date.now()-x) + "ms");
    console.log("result", html);
    return nightmare.end();
}).then(function(result) {
    …
}, function(err) {
   console.error(err); // notice that `throw`ing in here doesn't work
});
like image 87
Bergi Avatar answered Oct 17 '22 07:10

Bergi