Multi threaded curl handling multiple connections at the same time

Question

Is curl_multi interface spawning new threads internally to handle multiple requests concurrently? Is it equal to spawning threads manually and just using curl_easy handles? What is more performant. I need to make up to 1000 concurrent requests.

https://curl.haxx.se/libcurl/c/multithread.html

Is using curl_multi equal to the example in the link above?

From: https://curl.haxx.se/libcurl/c/libcurl-multi.html

Enable multiple simultaneous transfers in the same thread without making it complicated for the application.

What does it mean? How does it handle multiple transfers in the same thread? I could as well create 100 threads with 100 curl_easy handles and make requests there.

Maybe the question should be: When to use multiple threads and when to use curl_multi.

Answer 1

There's no easy or simple answer. libcurl allows you and your application to make the decision and supports working in either mode.

The libcurl multi interface is a single-core single-thread way to do a large amount of parallel transfers in the same thread. It allows for easy reuse of caches, connections and more. That has its clear advantages but will make it CPU-bound in that single CPU.

Doing multi-threaded transfers will make each thread/handle has its own cache and connection pool etc which will change when they will be useful, but will make the transfers less likely to be CPU-bound when you can spread them out over a larger set of cores/CPUs.

Which the right design decision is for you, is not easy for us to tell.

Answer 2

No all the connections on a single curl_multi handle operate on the same thread. It uses a single select/poll/epoll event loop and non-blocking sockets to process all the connection concurrently on the same thread.