Scraping ajax pages using python

Question

I've already seen this question about scraping ajax, but python isn't mentioned there. I considered using scrapy, i believe they have some docs on that subject, but as you can see the website is down. So i don't know what to do. I want to do the following:

I only have one url, example.com you go from page to page by clicking submit, the url doesn't change since they're using ajax to display the content. I want to scrape the content of each page, how to do it?

Lets say that i want to scrape only the numbers, is there anything other than scrapy that would do it? If not, would you give me a snippet on how to do it, just because their website is down so i can't reach the docs.

Answer 1

First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.

Speaking about handling ajax while web scraping. Basically, the idea is rather simple:

• open browser developer tools, network tab
• go to the target site
• click submit button and see what XHR request is going to the server
• simulate this XHR request in your spider

Also see:

• Can scrapy be used to scrape dynamic content from websites that are using AJAX?
• Pagination using scrapy

Hope that helps.

Answer 2

I found the answer very useful but I would like to make it more simple.

response = requests.post(request_url, data=payload, headers=request_headers)

request.post takes three parameters url, data and headers. Values for these three attributes can be found in the XHR request.

Copy the whole request header and form data to load into the above variables and you are good to go