Menu
There are few concurrency settings in Scrapy, like CONCURRENT_REQUESTS. Does it mean, that Scrapy crawler is multi-threaded? So if I run scrapy crawl my_crawler it will literally fire multiple simultaneous requests in parallel?
Im asking because, I've read that Scrapy is single-threaded.
272
Python is NOT a single-threaded language. Python processes typically use a single thread because of the GIL. Despite the GIL, libraries that perform computationally heavy tasks like numpy, scipy and pytorch utilise C-based implementations under the hood, allowing the use of multiple cores.
First, numpy supports multithreading, and this can give you a speed boost in multicore environments!
Multi-threaded capture and display. The following example implemented in tutorial-grabber-opencv-threaded. cpp shows how to implement a multi-threaded application, where image capture is executed in one thread and image display in an other one. The capture is here performed thanks to OpenCV cv::VideoCapture class.
Advantages of Multithreaded Processes All the threads of a process share its resources such as memory, data, files etc. A single application can have different threads within the same address space using resource sharing. It is more economical to use threads as they share the process resources.
Scrapy is single-threaded, except the interactive shell and some tests, see source.
It's built on top of Twisted, which is single-threaded too, and makes use of it's own asynchronous concurrency capabilities, such as twisted.internet.interfaces.IReactorThreads.callFromThread, see source.
181
Scrapy does most of it's work synchronously. However, the handling of requests is done asynchronously.
I suggest this page if you haven't already seen it.
http://doc.scrapy.org/en/latest/topics/architecture.html
edit: I realize now the question was about threading and not necessarily whether it's asynchronous or not. That link would still be a good read though :)
regarding your question about CONCURRENT_REQUESTS. This setting changes the number of requests that twisted will defer at once. Once that many requests have been started it will wait for some of them to finish before starting more.
42
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
