What is the difference between Scrapy's spider middleware and downloader middleware? [closed]

Question

Both middleware can process Request and Response. But what is the difference?

Answer 1

While they have almost identical interfaces, they serve different purposes:

• Downloader middlewares modify requests and responses or generate requests in response to responses. They don't directly interact with spiders. Some examples are middlewares that implement cookies, caching, proxies, redirects, setting user-agent headers, etc. They just add functionality to the downloader system.

• Spider middlewares modify things that pass in and out of spiders, like requests, items, exceptions, and start_requests. They do share some basic functionality with downloader middlewares, but they can't generate requests in response to responses. They stand between the spiders and the downloader. One example is filtering out responses with bad HTTP status codes.

Some middlewares can function as either a downloader middleware or a spider middleware, but they're often trivial and will be forced into one category or the other once you add more complex functionality.