My yet to be released Delphi 2010 application allows users to upload their files to my servers. Right now I'm using HTTPS POST to send the files, the (simplified) algorithm is basically:
ie. for a 1MB file:
--> Get Slice #1 (256KB)
--> Upload Slice #1 using TidHTTP.Post()
--> Get Slice #2 (256KB)
--> Upload Slice #2 using TidHTTP.Post()
--> Get Slice #3 (256KB)
--> Upload Slice #3 using TidHTTP.Post()
--> Get Slice #4 (256KB)
--> Upload Slice #4 using TidHTTP.Post()
I'm using Indy 10. I (ab)used my profiler over and over and there are not much left to optimize except changing the upload routine itself.
I'm also using multi-threading, and even though I did my best to optimize my code, my benchmarks still tell me I can do better (there are other well optimized software that do achieve a much better timing...almost twice as fast as my upload routine!)
I know it's not my server's fault...here are the ideas that I still need to explore:
I tried grouping slices in a single POST, naturally this resulted in a performance boost (20-35%) but resuming capability is now reduced.
I also thought about using SFTP / SSH, but I'm not sure if it's fast.
Use web sockets to implement resumable upload (like this component), I'm not sure about speed either.
Now my question is: is there something I can do to speed up my upload? I'm open to any suggestion that I can implement, including commandline tools (if license allows me to ship it with my application), provided that:
Also, because of major security concerns, FTP is a not something I'd want to implement.
Thanks a lot!
I would suggest doing a single TIdHTTP.Post()
for the entire file without chunking it at all. You can use the TIdHTTP.OnWork...
events to keep track of how many bytes were sent to the server so you know where to resume from if needed. When resuming, you can use the TIdHTTP.Request.CustomHeaders
property to include a custom header that tells the server where you are resuming from, so it can roll back its previous file to the specfiied offset before accepting the new data.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With