CasperJS can not trigger twitter infinite scroll

Question

I am trying to get some information from twitter using CasperJS. And I'm stuck with infinite scroll. The thing is that even using jquery to scroll the page down nothings seems to work. Neither scrolling, neither triggering the exact event on window (smth like uiNearTheBottom) doesn't seem to help. Interesting thing - all of these attempts work when injecting JS code via js console in FF & Chrome. Here's the example code :

casper.thenEvaluate(function(){
    $(window).trigger('uiNearTheBottom');
});

or

casper.thenEvaluate(function(){
    document.body.scrollTop  =  document.body.scrollHeight;
});

Answer 1

If casper.scrollToBottom() fails you or casper.scroll_to_bottom(), then the one below will serve you:

this.page.scrollPosition = { top: this.page.scrollPosition["top"] + document.body.scrollHeight, left: 0 };

A working example:

casper.start(url, function () {
 this.wait(10000, function () {
    this.page.scrollPosition = { top: this.page.scrollPosition["top"] + document.body.scrollHeight, left: 0 };
    if (this.visible("div.load-more")) {
        this.echo("I am here");
    }
})});

It uses the underlying PhantomJS scroll found here

Answer 2

CasperJs is based on PhantomJS and as per below discussion no window object exist for the headless browser.

You can check the discussion here

Answer 3

On Twitter you can use:

casper.scrollToBottom();
casper.wait(1000, function () {
    casper.capture("loadedContent.png");
});

But if you include jQuery... , the above code won't work!

var casper = require('casper').create({
    clientScripts: [
        'jquery-1.11.0.min.js'
    ]
});

The script injection blocks Twitter's infinite scroll from loading content. On BoingBoing.net, CasperJS scrollToBottom() works with jQuery without blocking. It really depends on the site.

However, you can inject jQuery after the content has loaded.

casper.scrollToBottom();
casper.wait(1000, function () {
    casper.capture("loadedContent.png");

    // Inject client-side jQuery library
    casper.options.clientScripts.push("jquery.js");

    // And use like so...
    var height = casper.evaluate(function () {
        return $(document).height();
    });
});