Scrapy runs all spiders at once. I want to only run one spider at a time. Scrapy crawl <spider>

Question

I am new to Scrapy and am trying to play around with the framework. What is really frustrating is that when I run "scrapy crawl (name of spider)" it runs every single spider in my "spiders" folder. So I either have to wait out all of the spiders running or comment out all the spiders except for the one I am working with. It is very annoying. How can I make it so that scrapy only runs one spider at a time?

Answer 1

You can run scrapy from your script (https://scrapy.readthedocs.io/en/latest/topics/practices.html#run-from-script), for example:

import scrapy
from scrapy.crawler import CrawlerProcess

class YourSpider(scrapy.Spider):
    # Your spider definition


process = CrawlerProcess()
process.crawl(YourSpider)
process.start()