Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

www::curl - how to upload (post) large files

I use WWW::Curl to upload files:

use WWW::Curl::Easy 4.14;
use WWW::Curl::Form;

my $url = 'http://example.com/backups/?sid=12313qwed323';
my $params = {
    name => 'upload',
    action => 'keep',
    backup1 => [ '/tmp/backup1.zip' ],   # 1st file for upload
};

my $form = WWW::Curl::Form->new();
foreach my $k (keys %{$params}) {
    if (ref $params->{$k}) {
        $form->formaddfile(@{$params->{$k}}[0], $k, 'multipart/form-data');
    } else {
        $form->formadd($k, $params->{$k});
    }
}

my $curl = WWW::Curl::Easy->new() or die $!; 
$curl->setopt(CURLOPT_HTTPPOST, $form);
$curl->setopt(CURLOPT_URL, $url);

my $body;   
$curl->setopt(CURLOPT_WRITEDATA, \$body);
my $retcode = $curl->perform();
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE); 

nothing special here and this code works well.

I want to upload large files and I don't want to preload everything in the memory. At least that is what I heard that libcurl is doing.

CURLOPT_READFUNCTION accepts callbacks which returns parts of the content. That means that I cannot use WWW::Curl::Form to set POST parameters but that I have to return the whole content through this callback. Is that right?

I think that the code could look like this:

use WWW::Curl::Easy 4.14;

my $url = 'http://example.com/backups/?sid=12313qwed323'
my $params = {
    name => 'upload',
    action => 'keep',
    backup1 => [ '/tmp/backup1.zip' ],   # 1st file for upload
};

my $fields;
foreach my $k (keys %{$params}) {
    $fields .= "$k=".(ref $params->{$k} ? '@'.@{$params->{$k}}[0] : uri_escape_utf8($params->{$k}))."&";
}
chop($fields);

my $curl = WWW::Curl::Easy->new() or die $!;
$curl->setopt(CURLOPT_POST, 1);
$curl->setopt(CURLOPT_POSTFIELDS, $fields); # is it needed with READFUNCTION??
$curl->setopt(CURLOPT_URL, $url);

my @header = ('Content-type: multipart/form-data', 'Transfer-Encoding: chunked');
$curl->setopt(CURLOPT_HTTPHEADER, \@header);

#$curl->setopt(CURLOPT_INFILESIZE, $size);
$curl->setopt(CURLOPT_READFUNCTION, sub {

    # which data to return here?
    # $params (without file) + file content?

    return 0;
});

Which data does CURLOPT_READFUNCTION callback have to return? $params + File(s) content? In which format?

Do I really have to create the data (returned by CURLOPT_READFUNCTION) by myself or is there a simple way to create it in the right format?

Thanks

like image 674
toktok Avatar asked Feb 28 '12 15:02

toktok


People also ask

How do I upload files using curl?

How to send a file using Curl? To upload a file, use the -d command-line option and begin data with the @ symbol. If you start the data with @, the rest should be the file's name from which Curl will read the data and send it to the server. Curl will use the file extension to send the correct MIME data type.

How do you upload a zip file using curl command?

zip file into a folder on your computer (i.e., C:\Curl). Invoke the CURL command from any folder, including the CURL installation folder (C:\Curl\bin), to your Windows PATH environment variable. Enter curl –version on the Windows command line to check if it was installed correctly and ensure you can use Curl commands.

How do I handle a large upload?

Possible solutions: 1) Configure maximum upload file size and memory limits for your server. 2) Upload large files in chunks. 3) Apply resumable file uploads. Chunking is the most commonly used method to avoid errors and increase speed.

How do you send a POST request on curl?

When the -F option is used, curl sends the data using the multipart/form-data Content-Type. Another way to make a POST request is to use the -d option. This causes curl to send the data using the application/x-www-form-urlencoded Content-Type.


2 Answers

Test 16formpost.t is relevant. As you can see, it's completely disabled. This fact and my fruitless experiments with various return values for the callback function lets me believe the CURLOPT_READFUNCTION feature is known broken in the Perl binding.

I have to return the whole content through this callback. Is that right?

No, you can feed it the request body piecewise, suitable for chunked encoding. The callback will be necessarily called several times, according to the limit set in CURLOPT_INFILESIZE.

Which data does CURLOPT_READFUNCTION callback have to return?

A HTTP request body. Since you do a file upload, this means Content-Type multipart/form-data. Following is an example using HTTP::Message. CURLOPT_HTTPPOST is another way to construct this format.

use HTTP::Request::Common qw(POST);
use WWW::Curl::Easy 4.14;

my $curl = WWW::Curl::Easy->new or die $!;
$curl->setopt(CURLOPT_POST, 1);
$curl->setopt(CURLOPT_URL, 'http://localhost:5000');
$curl->setopt(CURLOPT_HTTPHEADER, [
    'Content-type: multipart/form-data', 'Transfer-Encoding: chunked'
]);
$curl->setopt(CURLOPT_READFUNCTION, sub {
    return POST(undef, Content_Type => 'multipart/form-data', Content => [
        name    => 'upload',
        action  => 'keep',
        backup1 => [ '/tmp/backup1.zip' ],   # 1st file for upload
    ])->content;
});
my $r = $curl->perform;
like image 59
daxim Avatar answered Oct 19 '22 12:10

daxim


The CURLOPT_READFUNCTION callback is only used for chunked tranfer encoding. It may work, but I haven't been able to get it to and found that doing so wasn't necessary anyway.

My use case was for upload of data to AWS, where it's not ok to upload the data as multi-part form data. Instead, it's a straight POST of the data. It does require that you know how much data you're sending the server, though. This seems to work for me:

my $infile = 'file-to-upload.json';
my $size = -s $infile;
open( IN, $infile ) or die("Cannot open file - $infile. $! \n");

my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER,       1);
$curl->setopt(CURLOPT_NOPROGRESS,   1);
$curl->setopt(CURLOPT_POST,         1);
$curl->setopt(CURLOPT_URL,          $myPostUrl);
$curl->setopt(CURLOPT_HTTPHEADER,   
    ['Content-Type: application/json']); #For my use case
$curl->setopt(CURLOPT_POSTFIELDSIZE_LARGE, $size);
$curl->setopt(CURLOPT_READDATA, \*IN);

my $retcode = $curl->perform;

if ($retcode == 0) {
    print("File upload success\n");
} 
else {
    print("An error happened: $retcode ".$curl->strerror($retcode)."\n");
}

The key is providing an open filehandle reference to CURLOPT_READDATA. After that, the core curl library handles the reads from it without any need for callbacks.

like image 29
James Beck Avatar answered Oct 19 '22 13:10

James Beck