Something faster than get_headers()

Question

I'm trying to make a PHP script that will check the HTTP status of a website as fast as possible.

I'm currently using get_headers() and running it in a loop of 200 random urls from mysql database.

To check all 200 - it takes an average of 2m 48s.

Is there anything I can do to make it (much) faster?

(I know about fsockopen - It can check port 80 on 200 sites in 20s - but it's not the same as requesting the http status code because the server may responding on the port - but might not be loading websites correctly etc)

Here is the code..

<?php
  function get_httpcode($url) {
    $headers = get_headers($url, 0);
    // Return http status code
    return substr($headers[0], 9, 3);
  }

  ###
  ## Grab task and execute it
  ###


    // Loop through task
    while($data = mysql_fetch_assoc($sql)):

      $result = get_httpcode('http://'.$data['url']);   
      echo $data['url'].' = '.$result.'<br/>';

    endwhile;
?>

Answer 1

You can try CURL library. You can send multiple request parallel at same time with CURL_MULTI_EXEC

Example:

$ch = curl_init('http_url'); 
curl_setopt($ch, CURLOPT_HEADER, 1); 
$c = curl_exec($ch); 
$info = curl_getinfo($ch, CURLINFO_HTTP_CODE);
print_r($info);

UPDATED

Look this example. http://www.codediesel.com/php/parallel-curl-execution/

Answer 2

I don't know if this is an option that you can consider, but you could run all of them almost at the same using a fork, this way the script will take only a bit longer than one request http://www.php.net/manual/en/function.pcntl-fork.php

you could add this in a script that is ran in cli mode and launch all the requests at the same time, for example

Edit: you say that you have 200 calls to make, so a thing you might experience is the database connection loss. the problem is caused by the fact that the link is destroyed when the first script completes. to avoid that you could create a new connection for each child. I see that you are using the standard mysql_* functions so be sure to pass the 4th parameter to be sure you create a new link each time. also check the maximum number of simultaneous connections on your server