Call to URLFetch failed with application error 5 for url x

Question

Background:

I am using google app engine and am having a weird bug in my site crawler.

I have a backend that will automatically crawl a site every night. This is instigated by a task pushed to a pushQueue due to time limits in php.

Problem:

When I manually run the script that creates the task, the task completes as expected with no errors. However when cron launches the task I get the following error.

Call to URLFetch failed with application error 5 for url x

Code:

function url_get_contents ($Url) {
    global $retry;
    try {
        if (!function_exists('curl_init')){ 
            die('CURL is not installed!');
        }
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $Url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        $output = curl_exec($ch);
        curl_close($ch);
        return $output;
    } catch (Exception $e) {
        syslog(LOG_INFO, 'Caught exception: ',  $e->getMessage());
        if($retry > 0){
            $retry -= 1;
            return url_get_contents($Url);
        }
        else{
            return null;
        }
    }
}

Thanks to syslog I can see that the $url is fine which is driving me crazy as it works when the exact same script is launched manually not through cron.

How can I fix this?

Thanks in advance.

Answer 1

"application error 5" means that the request deadline was exceeded.

You can increase the deadline for the request by using the option CURLOPT_TIMEOUT, so you code might look something like:

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $Url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_TIMEOUT, 60);  // 60 second timeout
    $output = curl_exec($ch);

Answer 2

EDIT: curl is definitely installed, as you have a statement which checks it in the first place, so my answer no longer stands.

The CURL error list http://curl.haxx.se/libcurl/c/libcurl-errors.html says that error 5 is CURLE_COULDNT_RESOLVE_PROXY. Somehow your curl cannot access web, probably.

The curl is probably not enabled for your CLI environment.

Check out the configuration here:

php5/       phpmyadmin/ 
root@server1-webhost:~# cd /etc/php5
root@server1-webhost:/etc/php5# ls
apache2  cli  conf.d  mods-available
root@server1-webhost:/etc/php5# cd cli/
root@server1-webhost:/etc/php5/cli# ls
conf.d  php.ini
root@server1-webhost:/etc/php5/cli# cd conf.d
root@server1-webhost:/etc/php5/cli/conf.d# ls
10-pdo.ini   20-gd.ini      20-mssql.ini   20-mysql.ini      20-pdo_mysql.ini
20-curl.ini  20-mcrypt.ini  20-mysqli.ini  20-pdo_dblib.ini
root@server1-webhost:/etc/php5/cli/conf.d#