Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Terminate Scrapy if a condition is met

Tags:

python

scrapy

I have written a scraper using scrapy in python. It contains 100 start_urls.

I want to terminate the scraping process once a condition is met. ie terminate scraping of a particular div is found. By terminate I mean it should stop scraping all the urls .

Is it possible

like image 594
user2129794 Avatar asked May 27 '14 08:05

user2129794


1 Answers

What you're looking for is the CloseSpider exception.

Add the following line somewhere at the top of your source file:

from scrapy.exceptions import CloseSpider

And when you detect that your termination condition is met, simply do something like

        raise CloseSpider('termination condition met')

in your callback method (instead of returning or yielding an Item or Request).

Note that requests that are still in progress (HTTP request sent, response not yet received) will still be parsed. No new request will be processed though.

like image 102
paul trmbrth Avatar answered Oct 06 '22 00:10

paul trmbrth