Reading POST data in PHP from cUrl

Question

I am using cUrl in PHP to request from some external service.

Interesting enough, the server is responding with raw "multipart/form-data" instead of binary file data.

My website is using a shared hosting, therefore PECL HTTP is not an option.

Is there a way to parse this data with PHP?

Sample code:

$response = curl_exec($cUrl);

/* $response is raw "multipart/form-data" string

   --MIMEBoundaryurn_uuid_DDF2A2C71485B8C94C135176149950475371
   Content-Type: application/xop+xml; charset=utf-8; type="text/xml"
   Content-Transfer-Encoding: binary

   (xml data goes here)

   --MIMEBoundaryurn_uuid_DDF2A2C71485B8C94C135176149950475371
   Content-Type: application/zip
   Content-Transfer-Encoding: binary

   (binary file data goes here)

*/

EDIT: I tried piping the response to a localhost HTTP request, but the respond data is likely to exceed the allowed memory size in PHP process. Expending mem limit is not very practical, this action also dramatically reduces the server performance dramatically.

If there is no alternatives to the original question, you may suggest a way to handle very large POST requests, along with XML parsing, in terms of streams in PHP.

I know this would be hard, please comment. I am open for discussions.

Answer 1

if you need the zip file from the response I guess you could just write a tmp file to save the curl response to, and stream that as a workaround: Never tried that with multipart curls, but I guess it should work.

$fh = fopen('/tmp/foo', 'w'); 
$cUrl = curl_init('http://example.com/foo'); 
curl_setopt($cUrl, CURLOPT_FILE, $fh); // redirect output to filehandle
curl_exec($cUrl); 
curl_close($cUrl);
fclose($fh); // close filehandle or the file will be corrupted

if you do NOT need anything but the xml part of the response you might want to disable headers

curl_setopt($cUrl, CURLOPT_HEADER, FALSE);

and add option to only accept xml as a response

curl_setopt($cUrl, CURLOPT_HTTPHEADER, array('Accept: application/xml'));
//That's a workaround since there is no available curl option to do so but http allows that

[EDIT]

A Shot in the dark... can you test with these curlopt settings to see if modifiying these help anything

$headers = array (
    'Content-Type: multipart/form-data; boundary=' . $boundary,
    'Content-Length: ' . strlen($requestBody),
    'X-EBAY-API-COMPATIBILITY-LEVEL: ' . $compatLevel,  // API version
    'X-EBAY-API-DEV-NAME: ' . $devID,
    'X-EBAY-API-APP-NAME: ' . $appID,
    'X-EBAY-API-CERT-NAME: ' . $certID,
    'X-EBAY-API-CALL-NAME: ' . $verb,
    'X-EBAY-API-SITEID: ' . $siteID, 
    );

$cUrl = curl_init();
curl_setopt($cUrl, CURLOPT_URL, $serverUrl);
curl_setopt($cUrl, CURLOPT_TIMEOUT, 30 );
curl_setopt($cUrl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($cUrl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($cUrl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($cUrl, CURLOPT_POST, 1);
curl_setopt($cUrl, CURLOPT_POSTFIELDS, $requestBody);
curl_setopt($cUrl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($cUrl, CURLOPT_FAILONERROR, 0 );
curl_setopt($cUrl, CURLOPT_FOLLOWLOCATION, 1 );
curl_setopt($cUrl, CURLOPT_HEADER, 0 );
curl_setopt($cUrl, CURLOPT_USERAGENT, 'ebatns;xmlstyle;1.0' );
curl_setopt($cUrl, CURLOPT_HTTP_VERSION, 1 );      // HTTP version must be 1.0
$response = curl_exec($cUrl);

if ( !$response ) {
    print "curl error " . curl_errno($cUrl ) . PHP_EOL;
}
curl_close($cUrl);

[EDIT II]

This is just a try, as mentioned I cannot get my curled pages to respond with a multipart form data. So be gentle with me here ;)

$content_type = ""; //use last know content-type as a trigger
$tmp_cnt_file = "tmp/tmpfile";
$xml_response = ""; // this will hold the "usable" curl response
$hidx = 0; //header index.. counting the number of different headers received

function read_header($cUrl, $string)// this will be called once for every line of each header received
{ 
    global $content_type, $hidx;
    $length = strlen($string);
    if (preg_match('/Content-Type:(.*)/', $string, $match))
    {
        $content_type = $match[1];
        $hidx++;
    }
    /* 
    should set  $content_type to 'application/xop+xml; charset=utf-8; type="text/xml"' for the first 
    and to 'application/zip' for the second response body   

    echo "Header: $string<br />\n";
    */
    return $length;
}

function read_body($cUrl, $string)
{
    global $content_header, $xml_response, $tmp_cnt_file, $hidx;
    $length = strlen($string);
    if(stripos ( $content_type , "xml") !== false)
        $xml_response .= $string;
    elseif(stripos ($content_type, "zip") !== false)
    {
        $handle = fopen($tmp_cnt_file."-".$hidx.".zip", "a");
        fwrite($handle, $string);
        fclose($handle);
    }
    /*
    elseif {...} else{...}
    depending on your needs

    echo "Received $length bytes<br />\n";
    */
    return $length;
}

and of course set the proper curlopts

// Set callback function for header
curl_setopt($cUrl, CURLOPT_HEADERFUNCTION, 'read_header');
// Set callback function for body
curl_setopt($cUrl, CURLOPT_WRITEFUNCTION, 'read_body');

don't forget to NOT save the curl response to a variable because of the memory issues, hopefully all you need will be in the $xml_response above anyways.

//$response = curl_exec($cUrl);
curl_exec($cUrl);

And for parsing your code you can refer to $xml_response and the temp files you created starting with tmp/tmpfile-2 in this scenario. Again, I have not been able to test the code above in any way. So this might not work (but it should imho ;))

[EDIT III]

Say we want curl to write all incoming data directly to another (outgoing) stream, in this case a socket connection

I'm not sure if it is as easy as this:

$fs = fsockopen($host, $port, $errno, $errstr);
$cUrl = curl_init('http://example.com/foo'); 
curl_setopt($cUrl, CURLOPT_FILE, $fs); // redirect output to sockethandle
curl_exec($cUrl); 
curl_close($cUrl);
fclose($fs); // close handle

else we will have to use our known write and header functions with just a little trick

//first open the socket (before initiating curl)
$fs = fsockopen($host, $port, $errno, $errstr);
// now for the new callback function
function socket_pipe($cUrl, $string)
{ 
    global $fs;
    $length = strlen($string);
    fputs($fs, $string); // add NOTHING to the received line just send it to $fs; that was easy wasn't it?
    return $length;
}
// and of course for the CURLOPT part
// Set callback function for header
curl_setopt($cUrl, CURLOPT_HEADERFUNCTION, 'socket_pipe');
// Set the same callback function for body
curl_setopt($cUrl, CURLOPT_WRITEFUNCTION, 'socket_pipe');

// do not forget to 
fclose($fs); //when we're done

The thing is, not editing the result and simply piping it to $fs will make it necessary that apache is listening on a certain port which you then assign your script to. Or you will need to add ONE header line directly after fsockopen

fputs($fp, "POST $path HTTP/1.0\n"); //where path is your script of course