Python Scrapy: What is the difference between "runspider" and "crawl" commands?

Question

Can someone explain the difference between runspider and crawl commands? What are the contexts in which they should be used?

Answer 1

In the command:

scrapy crawl [options] <spider>

<spider> is the project name (defined in settings.py, as BOT_NAME).

And in the command:

scrapy runspider [options] <spider_file>

<spider_file> is the path to the file that contains the spider.

Otherwise, the options are the same:

Options
=======
--help, -h              show this help message and exit
-a NAME=VALUE           set spider argument (may be repeated)
--output=FILE, -o FILE  dump scraped items into FILE (use - for stdout)
--output-format=FORMAT, -t FORMAT
                        format to use for dumping items with -o

Global Options
--------------
--logfile=FILE          log file. if omitted stderr will be used
--loglevel=LEVEL, -L LEVEL
                        log level (default: DEBUG)
--nolog                 disable logging completely
--profile=FILE          write python cProfile stats to FILE
--lsprof=FILE           write lsprof profiling stats to FILE
--pidfile=FILE          write process ID to FILE
--set=NAME=VALUE, -s NAME=VALUE
                        set/override setting (may be repeated)
--pdb                   enable pdb on failure

Since runspider doesn't depend on the BOT_NAME parameter, depending on the way you are customising your scrapers, you might find runspider more flexible.

Answer 2

The main difference is that runspider does not need a project. That is, you can write a spider in a myspider.py file and call scrapy runspider myspider.py.

The crawl command requires a project in order to find the project's settings, load available spiders from SPIDER_MODULES settings, and lookup the spider by name.

If you need quick spider for a short task, then runspider has less boilerplate required.