Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Casperjs/Phantomjs - hangs when loading google plus main page

After successful login I'm trying to navigate to https://plus.google.com/u/0/?tab=wX but casperjs hangs.

The last output is:

[debug] [phantom] Navigation requested:
url=https://clients6.google.com/static/proxy.html?jsh=m;/_/scs/apps-static/_/js/k=oz.gapi.en.Z6gj5B0lzyA.O/m=__features__/am=IQ/rt=j/d=1/t=zcms/rs=AItRSTPU0_gqMrtQ831rDdqYv8Z1ZnxcbA#parent=https://tal kgadget.google.com&rpctoken=640385943, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested:
url=https://plus.google.com/hangouts/_/pre?hl=en&authuser=0, type=Other, willNav igate=true, isMainFrame=false
[debug] [phantom] Navigation requested:
url=https://plus.google.com/u/0/_/blank, type=Other, willNavigate=true, isMainFr ame=false

It seems like it hangs on hangouts iframes. How to block loading of certain internal URLs?

I tried:

// try to hide hangouts
casper.on('page.resource.requested', function (requestData, request) {
    if (requestData.url.indexOf('plus.google.com/hangouts') != -1) {
        this.echo('RESOURCE ABOTRED ' + requestData.url);
        request.abort();
        return;
    }
});

casper.on('navigation.requested', function (url, navigationType, navigationLocked, isMainFrame) {
    if (url.indexOf('plus.google.com/hangouts') != -1 || url.indexOf('talkgadget') != -1) {
        // this.echo('ALARM!!!' + url);
        willNavigate = false;
        // request.abort();
        return false;
    }
});

but its does not help at all as navigation requested is information event I cant block it!

Please advise.

It seem like google+ main page is completely protected from scraping with phantomjs.

like image 305
Albert Avatar asked Feb 06 '14 16:02

Albert


1 Answers

It appears this approach may work:

casper.on('navigation.requested', function (url, navigationType, navigationLocked, isMainFrame) {
    if (url.indexOf('plus.google.com/hangouts') != -1 || url.indexOf('facebook.com') != -1) {
      this.page.navigationLocked = true;
    }else{
      this.page.navigationLocked = false;
    }
});

Update: Appears this approach may prevent navigation of other concurrent steps, back to the drawing board...

like image 164
David Thomas Avatar answered Nov 06 '22 09:11

David Thomas