Scrapy follow javascript input button

Question

I have following input's on the page:

<input name="ct99" value="" id="ct99" class="GetData" type="submit">
<input name="ct92" value="" id="ct92" class="GetData" type="submit">
<input name="ct87" value="" id="ct87" class="GetData" type="submit">

class GetData display some click-able icon. When click on it new page is opened. Some JavaScript taking care of it. How can I follow this?

I'm already try code below just to see if scrapy follow inputs, but without success.

def parse(self, response):
    sel = Selector(response)

    links = sel.xpath("//input[@class='GetData']").extract()
    for data in links:
        yield scrapy.FormRequest.from_response(response,
            formdata={}, callback=self.after_click)


def after_click(self, response):
    url = response.url
    print '\nURL', url

Answer 1

There are two common ways to approach the problem:

• using browser developer tools (Network tab), inspect the requests sent when you click a particular button and then simulate this request using scrapy.Request or scrapy.FormRequest

• automate a browser using selenium: locate the button and click it, then grab the .page_source and instantiate a Selector instance, see samples here:

  • Scrapy with Selenium crawling but not scraping
  • How to use selenium along with scrapy to automate the process?