PHP - curl_exec hangs

Question

I am having a strange problem with the below php function. Unfortunately this is one of those special "Production only" case.

function requestPost($url, $data)
{
        set_time_limit(60);
        $output = array();
        $curlSession = curl_init();

        if($curlSession == false)
                syslog(LOG_INFO,"Falied to create a curl sessions");

        // Set the URL
        curl_setopt ($curlSession, CURLOPT_URL, $url);
        // No headers, please
        curl_setopt ($curlSession, CURLOPT_HEADER, 0);
        // It's a POST request
        curl_setopt ($curlSession, CURLOPT_POST, 1);
        // Set the fields for the POST
        curl_setopt ($curlSession, CURLOPT_POSTFIELDS, $data);
        // Return it direct, don't print it out
        curl_setopt($curlSession, CURLOPT_RETURNTRANSFER,1);
        // This connection will timeout in 30 seconds
        curl_setopt($curlSession, CURLOPT_TIMEOUT,30);
        //The next two lines must be present for the kit to work with newer version of cURL
        //You should remove them if you have any problems in earlier versions of cURL
        curl_setopt($curlSession, CURLOPT_SSL_VERIFYPEER, FALSE);
        curl_setopt($curlSession, CURLOPT_SSL_VERIFYHOST, 1);

        //Send the request and store the result in an array
        syslog(LOG_INFO,"base.php::requestPost() :  BEFORE SENDING CURL ");

        $rawresponse = curl_exec($curlSession);

}

PHP 5.3.6 (cli) (built: Mar 17 2011 21:19:28) 
curl Version          : 7.20.1
NSS Version           : 3.12.9
apr Version           : 1.4.5
Php Version           : 5.3.6

This seems to hang at curl_exec randomly.I am not a php developer so i have no idea where to begin.

I have noticed that when the curl requests are "hung" , on stopping the httpd daemon it attempts to send out these requests, so it might appear they are getting buffered or stuck in the httpd server. Any pointers would be appreciated.

Edit: I managed to get the stack trace out of the web server.

(gdb) where
#0  0x00c15416 in __kernel_vsyscall ()
#1  0x002f522c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00a73a9d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00b9b869 in PR_WaitCondVar () from /lib/libnspr4.so
#4  0x00809c0f in NSSRWLock_LockWrite_Util () from /usr/lib/libnssutil3.so
#5  0x05e9ae0e in ?? () from /usr/lib/libnss3.so
#6  0x05ebd014 in ?? () from /usr/lib/libnss3.so
#7  0x05ebd60d in ?? () from /usr/lib/libnss3.so
#8  0x05eb0506 in SECMOD_LoadModule () from /usr/lib/libnss3.so
#9  0x05eb047f in SECMOD_LoadModule () from /usr/lib/libnss3.so
#10 0x05e7c007 in ?? () from /usr/lib/libnss3.so
#11 0x05e7c95e in NSS_Initialize () from /usr/lib/libnss3.so
#12 0x008e8609 in ?? () from /usr/lib/libcurl.so.4
#13 0x008e9215 in Curl_nss_connect () from /usr/lib/libcurl.so.4
#14 0x008df9a3 in Curl_ssl_connect () from /usr/lib/libcurl.so.4
#15 0x008bc1fa in Curl_http_connect () from /usr/lib/libcurl.so.4
#16 0x008c44c6 in Curl_protocol_connect () from /usr/lib/libcurl.so.4
#17 0x008c5075 in ?? () from /usr/lib/libcurl.so.4
#18 0x008c58cf in Curl_async_resolved () from /usr/lib/libcurl.so.4
#19 0x008d1b2f in Curl_perform () from /usr/lib/libcurl.so.4
#20 0x008d2a74 in curl_easy_perform () from /usr/lib/libcurl.so.4
#21 0x006b4693 in ?? () from /usr/lib/php/modules/curl.so
#22 0x056e4ac9 in ?? () from /etc/httpd/modules/libphp5.so
#23 0x056bb87e in execute () from /etc/httpd/modules/libphp5.so
#24 0x05693a66 in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#25 0x05639cb6 in php_execute_script () from /etc/httpd/modules/libphp5.so
#26 0x057236b3 in ?? () from /etc/httpd/modules/libphp5.so
#27 0x00dc6421 in ap_run_handler ()
#28 0x00dca166 in ap_invoke_handler ()
#29 0x00dd6fa8 in ap_process_request ()
#30 0x00dd39e8 in ?? ()
#31 0x00dcec71 in ap_run_process_connection ()
#32 0x00ddc44a in ?? ()
#33 0x00ddc7ee in ?? ()
#34 0x00ddd793 in ap_mpm_run ()
#35 0x00db0ab2 in main ()

It appears to be stuck in a pthread_condition wait state. Does any one have a clue ? The same request is successful every single time via the command line; so it does point towards the libraries.

Edit:

Following is the call it never comes out from.

* About to connect() to live.sagepay.com port 443 (#0)
*   Trying x.x.x.x... * connected
* Connected to live.sagepay.com (x.x.x.x) port 443 (#0)
* warning: ignoring unsupported value (1) of ssl.verifyhost
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using SSL_RSA_WITH_RC4_128_MD5
* Server certificate:
*       subject: CN=live.sagepay.com,OU="Member, VeriSign Trust Network",OU=Authenticated by VeriSign,OU=Terms of use at www.verisign.co.uk/rpa (c)05,OU=Sage,O=Sage (UK) Limited,L=Newcastle Upon Tyne,ST=TYNE AND WEAR,C=GB,serialNumber=x.x.x,OID.x.x.x=Private Organization,OID.x.x.x.x=GB
*       start date: Mar 05 00:00:00 2011 GMT
*       expire date: Mar 04 23:59:59 2013 GMT
*       common name: live.sagepay.com
*       issuer: CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust Network,O="VeriSign, Inc.",C=US
> POST /gateway/service/vspserver-register.vsp HTTP/1.1^M
Host: live.sagepay.com^M
Accept: */*^M
Content-Length: 664^M
Content-Type: application/x-www-form-urlencoded^M
^M
< HTTP/1.1 200 OK^M
< Date: Wed, 15 Jun 2011 16:11:43 GMT^M
< Server: Microsoft-IIS/6.0^M
< X-Powered-By: ASP.NET^M
< Content-Language: en-GB^M
< Content-Length: 276^M
< Set-Cookie: NSC_wjq-tbhfqbz-dpn-ofx=xxxx;expires=Wed, 15-Jun-2011 16:48:26 GMT;path=/;secure;httponly^M
< ^M
* Connection #0 to host live.sagepay.com left intact

Edit: Problem was with NSS, recompiled curl with OpenSSL and a few more hoops for dependencies( libssh2) & it seems to be working fine so far.

Cheers!

Answer 1

add the following at the end your script to get the cause of failure

if( $rawresponse === false )
    syslog( LOG_INFO , "base.php::requestPost() : ".curl_error($curlSession) );

EDIT 1

It could be an internal problem of curl. Before all check all server runtimes are up to date ( php, php-curl and apache at least). Check all their logs..... Then I would recommand comparing results between several production environment or a dev/test environment.

Finally try to narrow down a minimal testcase that could reproduce your issue and publish full code for the test case.

Answer 2

The stack trace clearly shows that the break you did occur within the NSS crypto library that your libcurl is built to use for SSL.

libcurl has got a lot of NSS-related fixes since the libcurl version you use, and possibly you're not using the most up-to-date NSS either. I would strongly suggest that you consider upgrading to the latest and greatest before you pull all your hair out on this problem.

To debug this problem, I would recommend you try to repeat it with the curl command line from the same server, and make good use of the --trace-ascii command.