how to upload a file from node.js

Question

I found many posts when I queried for this problem, but they all refer to how to upload a file from your browser to a node.js server. I want to upload a file from node.js code to another server. I tried to write it based on my limited knowledge of node.js, but it doesn't work.

function (data) {
  var reqdata = 'file='+data;
  var request = http.request({
    host : HOST_NAME,
    port : HOST_PORT,
    path : PATH,
    method : 'POST',
    headers : {
      'Content-Type' : 'multipart/form-data',
      'Content-Length' : reqdata.length
    }
  }, function (response) {
      var data = '';
      response.on('data', function(chunk) {
        data += chunk.toString();
      });
      response.on('end', function() {
        console.log(data);
      });
    });

  request.write(reqdata+'\r\n\r\n');
  request.end();
})

The above function is called by other code that generates data.

I tried to upload same data file using curl -F "file=@<filepath>" and the upload is successful. But my code fails. The server returns an application specific error which hints that the uploaded file was invalid/corrupt.

I collected tcpdump data and analysed it in wireshark. The packet sent from my node.js code lacks the boundary required for the multipart data. I see this message in wireshark packet

The multipart dissector could not find the required boundary parameter.

Any idea how to accomplish this in node.js code?

Answer 1

jhcc's answer is almost there.

Having to come up with support for this in our tests, I tweaked it slightly.

Here's the modified version that works for us:

var boundaryKey = Math.random().toString(16); // random string
request.setHeader('Content-Type', 'multipart/form-data; boundary="'+boundaryKey+'"');
// the header for the one and only part (need to use CRLF here)
request.write( 
  '--' + boundaryKey + '\r\n'
  // use your file's mime type here, if known
  + 'Content-Type: application/octet-stream\r\n' 
  // "name" is the name of the form field
  // "filename" is the name of the original file
  + 'Content-Disposition: form-data; name="my_file"; filename="my_file.bin"\r\n'
  + 'Content-Transfer-Encoding: binary\r\n\r\n' 
);
fs.createReadStream('./my_file.bin', { bufferSize: 4 * 1024 })
  .on('end', function() {
    // mark the end of the one and only part
    request.end('\r\n--' + boundaryKey + '--'); 
  })
  // set "end" to false in the options so .end() isn't called on the request
  .pipe(request, { end: false }) // maybe write directly to the socket here?

Changes are:

• ReadableStream.pipe returns the piped-to stream, so end never gets called on that. Instead, wait for end on the file read stream.
• request.end puts the boundary on a new line.

Answer 2

Multipart is pretty complex, if you want to make it look like how a client usually handles "multipart/form-data", you have to do a few things. You first have to select a boundary key, this is usually a random string to mark the beginning and end of the parts, (in this case it would be only one part since you want to send a single file). Each part (or the one part) will need a header (initialized by the boundary key), setting the content-type, the name of the form field and the transfer encoding. Once the part(s) are completed, you need to mark the end of each part with the boundary key.

I've never worked with multipart, but I think this is how it could be done. Someone please correct me if I'm wrong:

var boundaryKey = Math.random().toString(16); // random string
request.setHeader('Content-Type', 'multipart/form-data; boundary="'+boundaryKey+'"');
// the header for the one and only part (need to use CRLF here)
request.write( 
  '--' + boundaryKey + '\r\n'
  // use your file's mime type here, if known
  + 'Content-Type: application/octet-stream\r\n' 
  // "name" is the name of the form field
  // "filename" is the name of the original file
  + 'Content-Disposition: form-data; name="my_file"; filename="my_file.bin"\r\n'
  + 'Content-Transfer-Encoding: binary\r\n\r\n' 
);
fs.createReadStream('./my_file.bin', { bufferSize: 4 * 1024 })
  // set "end" to false in the options so .end() isnt called on the request
  .pipe(request, { end: false }) // maybe write directly to the socket here?
  .on('end', function() {
    // mark the end of the one and only part
    request.end('--' + boundaryKey + '--'); 
  });

Again, I've never done this before, but I think that is how it could be accomplished. Maybe someone more knowledgable could provide some more insight.

If you wanted to send it as base64 or an encoding other than raw binary, you would have to do all the piping yourself. It will end up being more complicated, because you're going to have to be pausing the read stream and waiting for drain events on the request to make sure you don't use up all your memory (if it's not a big file you generally wouldn't have to worry about this though). EDIT: Actually, nevermind that, you could just set the encoding in the read stream options.

I'll be surprised if there isn't a Node module that does this already. Maybe someone more informed on the subject can help with the low-level details, but I think there should be a module around somewhere that does this.

Answer 3

As the error message states you are missing the boundary parameter. You need to add a random string to separate each file from the rest of the files/form-data.

Here is how a request could look like:

The content type:

Content-Type:multipart/form-data; boundary=----randomstring1337

The body:

------randomstring1337
Content-Disposition: form-data; name="file"; filename="thefile.txt"
Content-Type: application/octet-stream

[data goes here]

------randomstring1337--

Note that the -- in the beginning and end of of the random string in the body is significant. Those are part of the protocol.

More info here http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html

Answer 4

The fastest way I was able to do this, that worked, was using the request package. The code was well documented and it just worked.

(For my testing I wanted a JSON result and non-strict SSL - there are many other options...)

var url = "http://"; //you get the idea
var filePath = "/Users/me/Documents/file.csv"; //absolute path created elsewhere
var r = request.post( {
  url: url,
  json: true,
  strictSSL: false
}, function( err, res, data ) {
  //console.log( "Finished uploading a file" );
  expect( err ).to.not.be.ok();
  expect( data ).to.be.ok();
  //callback(); //mine was an async test
} );
var form = r.form();
form.append( 'csv', fs.createReadStream( filePath ) );