Fastest way to ping thousands of websites using PHP

Question

I'm currently pinging URLs using CURL + PHP. But in my script, a request is sent, then it waits until the response comes, then another request, ... If each response takes ~3s to come, in order to ping 10k links it takes more than 8 hours!

Is there a way to send multiple requests at once, like some kind of multi-threading?

Thank you.

Answer 1

USe the curl_multi_* functions available in curl. See http://www.php.net/manual/en/ref.curl.php

You must group the URLs in smaller sets: Adding all 10k links at once is not likely to work. So create a loop around the following code and use a subset of URLS (like 100) in the $urls variable.

$all = array();
$handle = curl_multi_init();
foreach ($urls as $url) {
    $all[$url] = curl_init();
    // Set curl options for $all[$url]
    curl_multi_add_handle($handle, $all[$url]);
}
$running = 0;
do {
    curl_multi_exec($handle, $running;);
} while ($running > 0);
foreach ($all as $url => $curl) {
    $content = curl_multi_getcontent($curl);
    // do something with $content
    curl_multi_remove_handle($handle, $curl);
}
curl_multi_close($handle);