does multipart/form-data sends the whole file data at one go or in a stream

Question

I have a requirement of uploading a large file over HTTP to a remote server. I am researching on how to send the data using multipart/form-data.

I have gone through How does HTTP file upload work? and understood how it separates the file data using boundaries.

I wanted to know whether all the file data is sent at one go or is streamed with several requests to the remote server.

Because if it is sent at one go, it is not possible to read the whole data at the remote server and write it to a file.

But if it streamed, how does the remote server parses the streamed data, write this streamed data to a file and redo the same thing till all the data is streamed.

Sorry if it a noob question, I am researching about it as well.

Maybe it is outside the scope of multipart/form-data and HTTP is itself taking care of.

Any help is appreciated.

Answer 1

I hope these help.

1. For HTTP file upload (all data sent through HTTP actually). Things are done with Stream API. Meaning your request headers arrives first and the body/payload comes latter chunk by chunk - stream. Read this blog post for a better insight of HTTP Anatomy. So in the end all data is stream, nothing is "at one go", and the stream is of course within one HTTP request.
2. Usually, when you work with backend frameworks (Express for example), they process the incomming data (body) for you through core modules. Giving the feel that data comes as one go but actually they're not. (an example of such modules are multer for file upload, express.json for json payload in ExpressJS)
3. If you're using form/multipart for large file. As pointed out here, the file size limit is dependant on the server, so it's based your config.
4. More problem arises when you increase the file size and take into account traffic, server resource. In this case, instead of using built in HTTP data parsing modules, you might need to use StreamAPI and make your own custom stream (a cool concept is stream.pipe).

You might want to look at or this gist, or my blog. Good luck.