Non Blocking Curl requests in PHP

Question

So I have this function for making non-blocking curl requests. It works fine on what I've tested so far (small amounts of requests). But I need this to scale up to thousands of requests (maybe max 10,000). My issue is that I don't want to run into issues with too many parallel requests running at once.

What would you suggest to rate-limit the requests? Usleep? Requests in batches? The function is below:

function poly_curl($requests){

            $queue = curl_multi_init(); 
            $curl_array = array(); 
            $count = 0;
            foreach($requests as $request) 
            { 
                $curl_array[$count] = curl_init($request); 
                curl_setopt($curl_array[$count], CURLOPT_RETURNTRANSFER, true); 
                curl_multi_add_handle($queue, $curl_array[$count]); 
                $count++;
            } 

            $running = NULL; 
            do { 

               curl_multi_exec($queue,$running); 

            } while($running > 0); 

            $res = array(); 
            $count = 0;
            foreach($requests as $request) 
            { 
                $res[$count] = curl_multi_getcontent($curl_array[$count]); 
                $count++;
            } 

            $count = 0;
            foreach($requests as $request){ 
                curl_multi_remove_handle($queue, $curl_array[$count]); 
                $count++;
            } 
            curl_multi_close($queue);        
            return $res; 
    }

Answer 1

I think curl_multi_exec is bad for this purpose, because even if you use batches in groups of 100, 99 request could be finished and still will have to wait for the last request completion.

But you need 100 parallel requests and when one finishes, another is immediately started. So you cannot use curl_multi_exec at all.

I would use normal producer-consumer algorithm with multiple (constant number) consumers with every consumer processing only one url. For example php-resque and COUNT=100 php resque.php