low priority http upload in .net

Question

I'm writing a program that uploads huge amounts of data and I need to limit it's interference with web browsing and other user activities.

The upload is composed of many large-ish files that are transferred individually, the connection must be a standard HTTP POST (I have no control of the server) and I need to control the HTTP headers (the server uses them for authentication and metadata)

It's important that the upload will resume full speed when the user is no longer using the internet because otherwise it will never finish (I expect it will need to run for a week or more at full speed to complete).

I want to solve this problem by somehow making my HTTP connection low priority, detecting open browser windows and slowing down does not solve the problem because (a) the user may be using a non-browser app (FTP, twitter client, e-mail, etc.) and (b) I don't want to slow down if there's an open idle web browser window.

I've found BITS but I think it's not relevant for me since I need it to be a standard HTTP POST.

I'm using .net 3.5, the program is written in C# and I'm currently using HttpWebRequest for the upload.

Clarification: I’m writing consumer software that will run on the customer’s personal computer at home. My beta testers complain that the internet is slow when they run my program (understandable, since I am using all their bandwidth) so I want to give higher priority to other programs so their internet is no longer slow.

There is no fancy network infrastructure that can prioritize packets on the network and no IT team to install and configure anything, I do expect most customers will have a cheap wireless router they got for free from their ISP

Answer 1

Simultaneously keep track of number of bytes your app sends and the total bytes sent on the network using the System.Net.NetworkInformation.IPv4InterfaceStatistics class' bytesSent Property at a given interval. Subtract the total bytes your app has sent in that interval from the total bytes sent on the network (during the same interval). If the difference is high enough to where you need to throttle your uploading then do so. Once the difference becomes small enough, crank up the uploading.

Answer 2

I hope you have the throttling end of the problem covered, so you only need to know two things:

• WHEN to reduce upload speed

and

• HOW MUCH to reduce it

Now, using other API-s is maybe fine, but why learn them and introduce all new kind of bug generators, when you can do something more simple.

Now - the problem with uploads with most internet connections (say ADSL) is that they consume precious UPLOAD that is extremely limited, say 256-512k. User probably report the problems because your upload is killing the requests they send, so they can't get any data back.

I am proposing that you do two things... to resolve WHEN problem, do this:

• create simple GET request for your web server
• measure time that is needed for the server to fully process and respond, by reading the response to the end
• use one cut-off value for 'busy' and 'free' connection, or measure times for each client and calculate their cut-off value
• if time is greater, you are busy, do throttling, else, go full speed
• you can repeat this as much as you can - for example every minute

To resolve HOW MUCH is another question. First, you should measure how much you can upload when nobody else is using the connection - your baseline upload speed per client. Then, using the WHEN technique, you can throttle down the upload until you get 'free' result that will work for their connection.

Hope I made myself clear - ask if anything needs clarification.

EDIT:

Few more thing to consider:

1. sending binary data through the POST is wasteful since everything is encoded in BASE64. You should inspect some other means of upload.
2. throttling against the web server could be problem because if you upload big chunk of data slowly, maybe server will timeout
3. if you combine 1+2, you could devise a plan that can splice the data into chunks, say 1KB or more, and send them one by one. It alone could solve your throttling problem, if you do it in serialized fashion.