How to RAW POST huge XML file with curl - PHP

Question

is there a way of doing

curl -X POST -H "Content-Type:application/xml" --data @myfile.xml http://example.com

but directly in PHP?

CURLOPT_PUT/CURLOPT_UPLOAD as well as file_get_contents as well as exec

are not a solutions as it must be POST and the file is huge so must be streamed.

Any ideas?

Answer 1

I had a similar problem trying to feed huge ingestion files to elasticsearch's bulk API from PHP, until I realized that the bulk API endpoint accepted PUT requests. Anyway, this code performs POST requests with huge files:

$curl = curl_init();
curl_setopt( $curl, CURLOPT_PUT, 1 );
curl_setopt( $curl, CURLOPT_INFILESIZE, filesize($tmpFile) );
curl_setopt( $curl, CURLOPT_INFILE, ($in=fopen($tmpFile, 'r')) );
curl_setopt( $curl, CURLOPT_CUSTOMREQUEST, 'POST' );
curl_setopt( $curl, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json' ] );
curl_setopt( $curl, CURLOPT_URL, $url );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1 );
$result = curl_exec($curl);
curl_close($curl);
fclose($in);

Here, $tmpFile is the full path of the file containing the request body.

NOTE: The important part is setting CURLOPT_CUSTOMREQUEST to 'POST' even though CURLOPT_PUT is set.

You'd have to adapt the Content-Type: header to whatever the server is expecting.

Using tcpdump, I could confirm the request method to be POST:

POST /my_index/_bulk HTTP/1.1
Host: 127.0.0.1:9200
Accept: */*
Content-Type: application/json
Content-Length: 167401
Expect: 100-continue

[...snip...]

I'm using packages libcurl3 (version 7.35.0-1ubuntu2.5) and php5-curl (version 5.5.9+dfsg-1ubuntu4.11) on Ubuntu 14.04.

Answer 2

Curl supports post, so I believe you are looking for something like this: Posting or Uploading Files Using Curl With PHP

// URL on which we have to post data
$url = "http://localhost/tutorials/post_action.php";
// Any other field you might want to catch
$post_data['name'] = "khan";
// File you want to upload/post
$post_data['file'] = "@c:/logs.log";

// Initialize cURL
$ch = curl_init();
// Set URL on which you want to post the Form and/or data
curl_setopt($ch, CURLOPT_URL, $url);
// Data+Files to be posted
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
// Pass TRUE or 1 if you want to wait for and catch the response against the request made
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// For Debug mode; shows up any error encountered during the operation
curl_setopt($ch, CURLOPT_VERBOSE, 1);
// Execute the request
$response = curl_exec($ch);

// Just for debug: to see response
echo $response;

(All credit for code to original linked author).

As to "huge", you'll want to be more specific - kb, mb, gb, tb? The additional problems will be related to how long a PHP script can stay alive without being auto-terminated, script memory usage (which may require things be handled in chunks instead of loading the whole file), etc.

EDIT: Ah, for RAW post I think you'll be needing this then: Raw POST using Curl in PHP