How to get response from scrapy.Request without callback?

Question

I want to send a request and wait for a response from the server in order to perform action-dependent actions. I write the following

resp = yield scrapy.Request(*kwargs)

and got None in resp. In documentation I find that need to use call_back function, but this function call after processing next commands. How to wait response from server?

Answer 1

I found the inline_requests module which has inline_requests decorator.

It solved my problem.

Answer 2

This isn't really how scrapy should be used, as waiting for a response is the same as using a callback. If you need to keep processing previous responses in conjunction with the new one, you can always pass and keep passing the response on the meta argument.

Now, to make this sometimes more readable you can also use scrapy-inline-requests which makes exactly the same as explained before under the hood, as it doesn't stop scrapy but makes the following request in order (same as doing a request after another with callbacks).

If using scrapy-inline-requests please be careful on making the methods to only be generators and also sending new requests or items when a new inline request is being processed.