Using Deferred in Scrapy downloadermiddleware

Question

I'm going to use some blocking code(this waiting for free proxy) in Scrapy downloadermiddleware. I was going to use this method

But it's realy doesn't work in downloadermiddleware, because the method process_request(self, request, spider) waiting for isinstance(response, (Response, Request))

How best to do this?

Answer 1

You can use twisted method "deferToThread" to run the blocking code without blocking the MainThread

from twisted.internet.threads import deferToThread

class DownloaderMiddleware:    
    def process_request(self, request, spider):
        return deferToThread(self.run_blocking_code_in_diffrent_thread, request, spider)

    def run_blocking_code_in_diffrent_thread(self,request, spider) -> HtmlResponse:
        print("Code will block here on a diffrent thread and wont stop MainThread")
        request.meta["proxy"] = get_proxy_blocking_call()
        return request