Crawling: Difference between "query string parameter" and "request payload"

Question

I am trying to crawl a ajax site using Scrapy, the url is http://www.target.com/p/bounty-select-a-size-white-paper-towels-12-mega-rolls/-/A-14920157#prodSlot=medium_1_2&term=bounty

My goal is to get the store id. I did that by checking all the XHR request in chrome developer tool and find the one with name ("v1?request_type=availability&key=.....") to be the one I want.

My questions are:

1. In developer tool, there is "Query string parameter" which seems to be the part after ? of the request url. There is also Request Payload section, which is a json. So which one should I use to send to the server? If i need request payload, how can I send a json file?

2. when i send the whole url to get json: https://api.target.com/available_to_promise_aggregator/v1?request_type=availability&key=q0jGNkIyuqUTYIlzZKoCfK6ugaNGSP8h

I get "Request method 'GET' not supported", so should I use POST instead or there is something wrong I did?

Answer 1

You must send the query string as part of the URL after a ?, as you guessed.

To include a JSON payload in a request, and send the request as a POST request, use the method and body parameters of the Request class.