Python scrapy, how to do a random delay between each request in a single spider?

Question

I have a spider where i want to have a delay between each request, but i don't want it to be a fixed number rather a random amount of time between two bounds, how can i do that ?

Answer 1

First of all, avoid to use time.sleep because it blocks Twisted reactor and thus eliminates all the advantages of Scrapy concurrency.

By default, Scrapy's DOWNLOAD_DELAY setting adds random delay in range between 0.5 * DOWNLOAD_DELAY and 1.5 * DOWNLOAD_DELAY seconds. This behaviour manages by RANDOMIZE_DOWNLOAD_DELAY setting which is true by default.

Docs: https://doc.scrapy.org/en/latest/topics/settings.html#std:setting-DOWNLOAD_DELAY