Scrapy FormRequest.from_response() method

Question

Im am try to parse this page using Scrapy In order to show hidden text with price I have type the any zip code or random numbers in the field:

<input aria-label="Enter your zip code" role="textbox" name="searchTerm" class="form-control js-list-zip-entry-input" placeholder="ZIP Code" autocompletetype="find-a-store-search" tabindex="-1" type="text">

For response url with prices I try yse FormRequest.from_response , but it is not successfully. Maybe someone can explain how I can response with prices ?

import scrapy
from scrapy.http.request import Request
from scrapy.http import FormRequest
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class SattySpider(scrapy.Spider):
    name = "lowes-faucet"
    allowed_domains = ["lowes.com"]
    start_urls = [ "http://www.lowes.com/search?searchTerm=faucets"]

    def parse(self, response):
        yield FormRequest.from_response(response,
                                          formnumber=1,
                                          formxpath='id("store-locator-form")',
                                          formdata={'searchTerm': '58000'},
                                          callback=self.parse1, method="GET")
    def parse(self, response):
        open_in_browser(response)
        ...

Answer 1

The problem here is that the form is executing some javascript code before sending the actual request, and the request is being done to a different site, this one for example (you'll have to check which parameters to actually use). but of course passing more stuff, mostly cookies.

After doing that request, the server knows and sets that the current session is from that location, so you need to do another request to the previous site to get the information with the actual data.

In Scrapy it would be something like:

• Request to http://www.lowes.com/search?searchTerm=faucets
• Request to the previous link with cookies.
• Request again to http://www.lowes.com/search?searchTerm=faucets passing all the cookies

Scrapy should handle the cookies by itself, but of course you can't be 100% sure, there could be some custom cookies.