Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I merge results from target page to current page in scrapy?

Tags:

Need example in scrapy on how to get a link from one page, then follow this link, get more info from the linked page, and merge back with some data from first page.

like image 733
Jas Avatar asked Dec 11 '11 21:12

Jas


1 Answers

Partially fill your item on the first page, and the put it in your request's meta. When the callback for the next page is called, it can take the partially filled request, put more data into it, and then return it.

like image 183
Acorn Avatar answered Oct 05 '22 11:10

Acorn