Is it possible to upload a file with cURL from a pipe?

Question

I mean POSTing a standard file upload form. Usual command line contains this switch in this case:

-F "[email protected]"

However when I try to feed a named pipe made by linux command "mkfifo", eg. "mkfifo filename.zip", I always get an error message on the producer side:

curl: (23) Failed writing body (1856 != 16384)

And also some error message appears at consumer side of the fifo. I fed my fifo with another curl command on producer side, eg.:

 curl http://example.com/archive.zip > filename.zip

And on consumer side:

curl http://example.com/fileupload.php -F "[email protected]"

When I pass a Content-Length HTTP header at the consumer side of my fifo, I don't get error message at the producer side, but error message still appears at the consumer (uploading) side, unsuccessful upload.

 curl http://example.com/fileupload.php -F "[email protected]" -H "Content-Length: 393594678"

I also tried feeding cURL file upload a non-named pipe, causing cURL to read data from stdin (marked as @-), like:

   curl -# http://example.com/archive.zip | curl -# http://example.com/fileupload.php -F "file=@-"

In this case upload is successful, however the downloading and uploading progress are not in sync, I can see too separate hashmark progress indicators, one for the download and one for the upload, rather consecutively and not operating at the same time. On the top of that remote file is always named as "-", but this is not an issue for me, can be renamed later.

Further notice: I tried the aboves from a Ruby command line IRB / Pry session, and I have noticed that when I used Ruby command "system" to call the piped construct:

system %Q{curl -# http://example.com/archive.zip | curl -# http://example.com/fileupload.php -F "file=@-"}

I can see only one hashmark progress indicator in this case, so I think piping works as it should be, but I can see two consecutive hashmark progress indicator in this second case like this:

%x{curl -# http://example.com/archive.zip | curl -# http://example.com/fileupload.php -F "file=@-"}

Answer 1

Yes it is possible!

By default, curl will check all the provided arguments, figure out the size of all involved components (including the files) and send them in the constructed POST request. That means curl will check the size of the local file, which thus breaks when you use a fifo. Thus you need to do something about it!

chunked for fifo

By telling curl it should do the POST with chunked encoding instead of providing the full size ahead of time, curl will instead read the file in a streaming manner and just allow it to turn up to be whatever size it needs to be at the time the file (fifo) is read.

You can do this by setting the chunked header, which is used by curl as a signal to do the request chunked.

curl -H "Tranfer-Encoding: chunked" -F file=@fifo https://example.com

caveat

The reason this isn't the default behavior by curl is that this requires that the receiver is using HTTP/1.1 or later (which curl doesn't know until it gets the response back from the server). Old HTTP/1.0 servers don't speak "chunk".

formpost from stdin

When doing a formpost from stdin, curl will read the entire file off stdin first in memory, before doing the POST, to get the size of the content so that it can include that in the POST request.