Posting Gzipped data with curl

Question

im trying to use the system curl to post gzipped data to a server but i keep ending up with strange errors

`curl -sS -X POST -H "Content-Type: application/gzip" --data-binary $data $url`

gives

curl: no URL specified!

and

`curl -sS -X POST -H "Content-Type: application/gzip" --data-binary "$data" $url`

gives

sh: -c: line 0: unexpected EOF while looking for matching `"'
sh: -c: line 1: syntax error: unexpected end of file

Answer 1

Adding the " is a step in the right direction, but you didn't consider that $data might contains ", $, etc. You could use String::ShellQuote to address the issue.

use String::ShellQuote qw( shell_quote );

my $cmd = shell_quote(
   curl => (
      '-sS',
      '-X' => 'POST',
      '-H' => 'Content-Type: application/gzip',
      '--data-binary' => $data,
      $url,
   ),
);

my $output = `$cmd`;

Or you could avoid the shell entirely.

my @cmd = (
   curl => (
      '-sS',
      '-X' => 'POST',
      '-H' => 'Content-Type: application/gzip',
      '--data-binary' => $data,
      $url,
   ),
);

open(my $pipe, '-|', @cmd) or die $!;
my $output = do { local $/; <$pipe> };
close($pipe);

Or if you didn't actually need to capture the output, the following also avoids the shell entirely:

system(
   curl => (
      '-sS',
      '-X' => 'POST',
      '-H' => 'Content-Type: application/gzip',
      '--data-binary' => $data,
      $url,
   ),
);

That said, I don't see how you can possibly send strings containing NUL bytes, something a gzipped file is likely to have. I think your approach is inherently flawed.

Do you know that libcurl (the guts of curl) can be accessed via Net::Curl::Easy?

Answer 2

I did not succeed in getting curl to read the data straight from stdin, but process substitution did work, for example:

curl -sS -X POST -H "Content-Type: application/gzip" --data-binary @<(echo "Uncompressed data" | gzip) $url

This technique removes any need to having to write to a temporary file first.