Scrapyd: How to set scrapyd task priority?

Question

I have several scrapy projects. I deploy all of them with scrapyd. Some of the spiders are slow while others are fast. Now, I want to run fast spiders first. How to do?

Answer 1

I ran into the same issue. But I read the source code of scrapyd and it seems that scrapyd can handle it. This is not documented, however.

You need to add an argument named priority with the desired value. The default priority is 0.

For instance:

curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d priority=1