Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Resetting paused scrape, Scrapy

I know that with the command:

scrapy crawl somespider -s JOBDIR=crawls/somespider-1

I can pause/resume scrapes with CTRL+C.

What I want to know is how I can reset the scrapy and start back at the beginning. Is there a file I need to delete/empty?

M.

like image 691
Mats Avatar asked Apr 30 '16 14:04

Mats


1 Answers

Yes, you should remove your JOBDIR

scrapy crawl somespider -s JOBDIR=crawls/somespider-1
<ctr+C>
rm -rf crawls/somespider-1
like image 86
Daniil Mashkin Avatar answered Nov 04 '22 04:11

Daniil Mashkin