How to retry for 404 link not found in scrapy?

Question

Some sites give 404 error temporarily. But I paste in on browser it works. How to tell scrapy to retry the 404 status code links for 5 times.

Answer 1

There are two Scrapy settings relevant to what you need:

• RETRY_HTTP_CODES: you should override the default value in your project to include 404
• RETRY_TIMES: just set it to 5