How does HTTP file upload work for large files?

Question

I just want to elaborate this question: How does HTTP file upload work?. This is the form from the question:

<form enctype="multipart/form-data" action="http://localhost:3000/upload?upload_progress_id=12344" method="POST">
<input type="hidden" name="MAX_FILE_SIZE" value="100000" />
Choose a file to upload: <input name="uploadedfile" type="file" /><br />
<input type="submit" value="Upload File" />
</form>

What happens when the file is really large (i.e. 10 GB)? Does the browser put all the data into 1 request then send it to the server? How does the browser read the file and build the request(s) when it has only 2 GB of RAM?

Let say the file is a CSV. Assuming the server has little RAM and Disk space. Is there a way to stream the file to the server so that the server can parse each line instead of keeping the whole file in its RAM or Disk?

Detailed explanations are much appreciated (HTTP, TCP, etc.)

Answer 1

You can find some details here:

Large file upload though html form (more than 2 GB)

In a nutshell, the remote http server is what determines the maximum size of the HTTP POST. Anything larger than its configured maximum size, and you will receive an error.

Developing your own solution is an option. For your example of a large CSV file, imagine an HTTP POST that sends 1 line of the file. 100 lines would require 100 HTTP POSTs.

A major disadvantage is that an HTML form could not be used directly. Perhaps JavaScript could handle the posts for you, such as a variation of this post:

How to upload string as file with jQuery or other js framework

You would have some work to do on both sides if ordering is important (i.e. send some sort of sequence number as part of the form data).

There are other ways of accomplishing this. But hopefully the above suggestions will get you thinking.