CURL stop working

Question

I wanna search some links about a word from a list. So i am making a script :

//html code here.
<?
if (array_key_exists('form_action', $_POST)){
$pel=$_POST['url'];
$toplist=file_get_contents($pel);
$listgrabbing=explode("\r\n",$toplist);
foreach($listgrabbing as $item)
{    

$useragent="Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1; .NET CLR 1.1.4322; Alexa Toolbar; .NET CLR 2.0.50727)";
$urlto=$item;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urlto);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_COOKIEJAR, "COOKIE.txt"); 
curl_setopt($ch, CURLOPT_COOKIEFILE, "COOKIE.txt"); 
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,10); 
$buffer = curl_exec($ch);
$po = strpos($buffer,"article");
if ($po===false)
{
echo ($item."---Word didn't found!");
echo "<br>";
}
else {
echo ($item."---Word Found!");
echo "<br>";
}
}
}
?>

It is working fine. But sometimes script stop working suddenly. I don't know why. May it goes into a site which is not responding. but for this i used CURLOPT_CONNECTTIMEOUT . But i haven't find what is the wrong in script.

Actually my problem is , script is stop suddenly while running.

Answer 1

Try the the options CURLOPT_LOW_SPEED_TIME together with CURLOPT_LOW_SPEED_LIMIT

// the download speed must be at least 1 byte per second
curl_setopt(CURLOPT_LOW_SPEED_LIMIT, 1);
// if the download speed is below 1 byte per second for
// more than 30 seconds curl will give up
curl_setopt(CURLOPT_LOW_SPEED_TIME, 30);

This will prevent curl from 'hanging' on slow or dead connections if for a given timeout the download rate is below a given threshold. When the timeout is reached you can retry it or skip the url:

// skips the url if errors on download
$buffer = curl_exec($ch);
if ($buffer === FALSE) { 
    echo curl_error($ch);
    continue;
}

'Stop working' can have several reasons. The simplest is, the remote server crashed during its response without sending aTCP FIN. (I have seen this in the wild). So the underlying TCP connection wouldn't be closed and curl waits - forever - for the remaining bytes.

Also a firewall rule that blocks the port during transfer after the connection has been established might be the reason. Not so likely but also seen in the wild.

Another reason I can imagine is, that the remote server calculates the wrong 'Content-Length' HTTP header. Together with HTTP/1.1's 'Connection: keep-alive' this might make curl 'hanging' while waiting for remaing bytes that will never being send. To prevent from this you should explicitly use the header 'Connection: close'. This can be done as follows:

curl_setopt(CURLOPT_HTTPHEADER, array('Connection: close'));

However my suggestions are just workarounds to prevent your script from hanging. If you want to understand why curl hangs you'll have to trace network traffic. You can use Wireshark for it.