Scraping Google Keyword Tools with CasperJS and PhantomJS

Question

I'm currently trying to scrape Google Keyword Tools with CasperJS and PhantomJS (both excellent tools, thanks n1k0 and Ariya), but I can't get it to work.

Here is my current process:

1. Log in with my Google Account (to avoid captchas in the Keyword Tools).
2. Navigate to the Keyword Tools page.
3. Fill in the search form and press Search.

I'm stuck at step 3: the search form is not a regular HTML form, I can't use Casper#fill(), so instead I'm accessing the fields directly. Here are some of the syntaxes I tried to change the value of the Word or phrase field:

this.evaluate(function() {

    // Trying to change the value...
    document.querySelector('textarea.sP3.sBFB').value = 'MY SUPER KEYWORDS';
    document.querySelector('textarea.sP3.sBFB').setAttribute('value', 'MY SUPER KEYWORDS');
    document.querySelector('textarea').value = 'MY SUPER KEYWORDS';   // there's only one <textarea> on the page

    // Trying to change other attributes...
    document.querySelector('textarea.sP3.sBFB').textContent = 'MY SUPER KEYWORDS';
    document.querySelector('textarea').style.backgroundColor = 'yellow';
});

Nothing works. I'm doing a Casper#capture() right after to see what the field contains. As you can see, it confirms I am on the right page and that I am logged in, but the <textarea> is empty.

Strangely, I can access other parts of the DOM: I could change the text of a link that said Advanced Options and Filters to ___VINCE SAYS HELLO___ (see capture), by doing the following:

this.evaluate(function() {
    document.querySelector('a.sLAB').textContent = '___VINCE SAYS HELLO___';
});

PS. I know scraping Google Keyword Tools is against the TOS, but I'm thinking this question might be of interest to anyone trying to scrape a JavaScript/Ajax-heavy site.

Answer 1

document.querySelector('textarea.sP3.sBFB').value = 'MY SUPER KEYWORDS';

You can't use elt.value on a textarea. Did you try with elt.textContent?