Reliable and fast way to transfer large files over the internet

Question

I'm working with a setup involving many clients PCs and some server machines. I need to organize a reliable and fast method of file transfer between these PCs, that will be initiated by C# apps running on both. Any client may want to send/receive data from any server. The options are:

• FTP - Use FtpWebRequest or SOSFTP to upload files onto an FTP server. The server checks for new files on its filesystem and does the required steps.

• SCP - Secure file transfer. Usage same as FTP but increased security between machines. Probably slower than FTP.

• DropBox / Box.Net - Use an online cloud storage solution with a library such as SharpBox. May be free / paid. May be less secure considering the said party has your files.

• UDP - Use a library such as EME or GoAnywhere to transfer data from PC to PC directly via UDP. Probably faster but probably more unreliable since it uses custom technology.

What do you recommend?

Answer 1

I have a super-biased opinion, coming from a company that makes file transfer software, so I'm putting the disclaimer front and center to take everything I say with a grain of salt. ;-)

If you are sharing very large files to multiple endpoints, you are well served with one of two things:

1. An MFT (Managed File Transfer) solution
2. A product or API that uses a UDP-based protocol

1 - The MFT solutions typically use TCP-based transfer (like FTP) as the transport mechanism, but usually include additional reliability mechanisms not found in pure TCP. They will also include tools to schedule and organize transfers between many endpoints. To the best of my knowledge, there are not any significant MFT solutions that are available for free or low cost, but there are a plethora of enterprise-grade MFT solutions out there at the expected costs.

2 - For several years now, some companies have built proprietary control and reliability mechanisms into protocols built on top of UDP. By creating this custom layer, UDP (which has no reliability of its own) actually becomes MORE reliable and can have more features than TCP-based transfer can ever have, while not being adversely affected by packet loss and latency (read as: typically much faster transfer speeds). This protocol is not necessarily independent of MFT, either (it will inherently have some MFT characteristics and vendors might bake it into an MFT solution), but is potentially usable for its raw protocol (via API) and without all the UI trappings of a traditional "MFT solution".

There ARE open-source UDP-based file transfer protocols out there like Tsunami or UDT. They are not particularly well-maintained or feature-rich (there's some of my bias!) and as far as I know, none have a native C# library. On the same note, though, none of the commercial offerings that I'm aware of have a native C# library either, though wrappers are available.

---

If the kinds of transfers you describe are going to be a regular and ongoing part of the organization's daily activities, I have a hard time recommending anything beyond what I mention above. And to do it "right" (or at least without a non-trivial development project) you will probably want a commercial solution.

Answer 2

I found it interesting that GoAnywhere is mentioned here. It is considered an MFT solution. I've used MFT for several years and found it very effective in doing exactly what you are describing here without having to write C# code. It's not free but I think it is well worth the cost.

It allows you to automate and schedule the various transfer protocols such as FTP, SFTP, FTPS, PGP, HTTPS, and also across network connections to multiple shared pc's and servers. MFT also allows me to work on formatting changes if I need to massage the data to work with different applications. Run queries to extract just the data I need from various data sources. It also logs all transactions so that I can verify that it has sent the files and can be setup to alert me with an email if there are any errors. I found it to be the perfect tool for all my data transfer requirements.

I am NOT an employee of any MFT software companies but a very happy customer that loves the product. It has made my work life SO much easier. There's a great example of how it helps IT staff here at http://blog.linomasoftware.com/2012/01/24/managed-file-transfer-changed-my-life/