Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Uploading big files over HTTP

Tags:

c++

http

php

upload

I need to upload potentially big (as in, 10's to 100's of megabytes) files from a desktop application to a server. The server code is written in PHP, the desktop application in C++/MFC. I want to be able to resume file uploads when the upload fails halfway through because this software will be used over unreliable connections. What are my options? I've found a number of HTTP upload components for C++, such as http://www.chilkatsoft.com/refdoc/vcCkUploadRef.html which looks excellent, but it doesn't seem to handle 'resume' of half done uploads (I assume this is because HTTP 1.1 doesn't support it). I've also looked at the BITS service but for uploads it requires an IIS server. So far my only option seems to be to cut up the file I want to upload into smaller pieces (say 1 meg each), upload them all to the server, reassemble them with PHP and run a checksum to see if everything went ok. To resume, I'd need to have some form of 'handshake' at the beginning of the upload to find out which pieces are already on the server. Will I have to code this by hand or does anyone know of a library that does all this for me, or maybe even a completely different solution? I'd rather not switch to another protocol that supports resume natively for maintenance reasons (potential problems with firewalls etc.)

like image 781
Roel Avatar asked Jan 29 '09 16:01

Roel


People also ask

How can I upload a large file over HTTP?

Use HTML5 File API to upload large files. which has the concept of slicing, so that you can upload large files.

How does HTTP deliver a large file?

Send chunked data In version 1.1, HTTP introduced chunked data to help with the large-data cases. When sending a response, the server adds a header Transfer-Encoding: chunked , letting the browser know that data is transmitted in chunks.

Can you send files over HTTP?

An HTTP file transfer is the process of transferring a file between multiple nodes/devices using the HTTP protocol, or more generally, the Internet. It is one of the most commonly used methods for sending, receiving or exchanging data and files over the Internet or a TCP/IP-based network.


2 Answers

I'm eight months late, but I just stumbled upon this question and was surprised that webDAV wasn't mentioned. You could use the HTTP PUT method to upload, and include a Content-Range header to handle resuming and such. A HEAD request would tell you if the file already exists and how big it is. So perhaps something like this:

1) HEAD the remote file

2) If it exists and size == local size, upload is already done

3) If size < local size, add a Content-Range header to request and seek to the appropriate location in local file.

4) Make PUT request to upload the file (or portion of the file, if resuming)

5) If connection fails during PUT request, start over with step 1

You can also list (PROPFIND) and rename (MOVE) files, and create directories (MKCOL) with dav.

I believe both Apache and Lighttpd have dav extensions.

like image 62
ryan Avatar answered Sep 28 '22 09:09

ryan


You need a standard size (say 256k). If your file "abc.txt", uploaded by user x is 78.3MB it would be 313 full chunks and one smaller chunk.

  1. You send a request to upload stating filename and size, as well as number of initial threads.
  2. your php code will create a temp folder named after the IP address and filename,
  3. Your app can then use MULTIPLE connections to send the data in different threads, so you could be sending chunks 1,111,212,313 at the same time (with separate checksums).
  4. your php code saves them to different files and confirms reception after validating the checksum, giving the number of a new chunk to send, or to stop with this thread.
  5. After all thread are finished, you would ask the php to join all the files, if something is missing, it would goto 3

You could increase or decrease the number of threads at will, since the app is controlling the sending.

You can easily show a progress indicator, either a simple progress bar, or something close to downthemall's detailed view of chunks.

like image 36
Osama Al-Maadeed Avatar answered Sep 28 '22 08:09

Osama Al-Maadeed