How to FTP constantly incoming files

Question

Ok, here's the situation... I have an application that generates about 8 files per second. Each file is 19-24kb. This generates about 10 to 11 MB per minute. This question is not about how to ftp, because I have that solution already... The question is more about how to keep up with the flow of data (only a 2mb upload bandwidth in most cases, unless I am travelling to a client site that has a large pipe). I dont care if ftp takes longer to transfer then the rate of flow, but I want to know if anyone has an idea on how to batch the files to move them so that when the ftp process is finished it will delete just those files it transfered and then move on to the next batch. Here is what I was thinking:

Multi thread the app, first thread runs the app, second thread is a timer that creates a text file every 'N' minutes with all the files created in that time span. StreamRead the file and move the files that are in text to another location (maybe create a temp folder) and then ftp those files, then delete files, folder and textfile... in the mean time, more text files are being written and temp folders being created. Does this sound feasible? I will take any suggestions that anyone has under advisement, just looking for the fastest and most reliable path.

Please dont ask to see the code, there is no reason to see it considering we are working with hypotheticals.

Answer 1

I would create a service and add the incoming files into a concurrent collection using FileSystemWatcher, System.Threading.Timer or both (FileSystemWatcher may miss files if its buffer is overrun so it is a good idea to have a timer going to pick up any files that are missed). When files come in I would move them into a separate folder and would process them using .NET 4.0 tasks. I would then do any necessary post processing in continuation steps to the original tasks. You can have continuation steps that handle any faults and different continuation steps that occur upon success. Each of these tasks will spin up a thread in the thread pool and will be managed for you.

Here is an example from http://msdn.microsoft.com/en-us/library/dd997415.aspx of a OnlyOnFaulted continuation task. You could have a second continuation task that will only run when successful.

var task1 = Task.Factory.StartNew(() =>
{
    throw new MyCustomException("Task1 faulted.");
})
.ContinueWith((t) =>
    {
        Console.WriteLine("I have observed a {0}",
            t.Exception.InnerException.GetType().Name);
    },
    TaskContinuationOptions.OnlyOnFaulted);