How do I save files to hard disk in a separate thread?

Question

I've a camera and I'm reading the images in real time into an array. I'm applying some algorithm to the image and displaying it. Then I get the next image and display it as well. So I'm streaming images from the camera to the display. However I also want to save images to hard disk once I've displayed them. I tried using the main thread but everything slowed down too much. I then tried using ThreadPool (see code below). This doesn't slow the display down but I've found the images aren't being saved properly. It looks like they are not in the expected order and after about 50 images have been saved the subsequent image data looks garbled. I'm guessing too many threads are being started.

Is there a better way to do this? I think I only need one thread to save the images. Maybe some kind of queue that saves each image sequentially. Just as long as its done in the background and doesn't slow down the display. If someone could post a code snippet that would be fantastic.

short[] image1 = new short[20000];
while(streaming)
{
    ReadImageFromCamera(ref image1)
    ImageData data;    

    data.fileName = imageNumber;
    data.image = image1;

    ThreadPool.QueueUserWorkItem(WriteImageToFile, data);  // Send the writes to the queue
}


private void WriteImageToFile(object imageData) {

    try {
        ImageData data = (ImageData)imageData;
        System.Runtime.Serialization.Formatters.Binary.BinaryFormatter bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();

        string fName = myDirectory + @"/" + Convert.ToString(data.fileName) + @".spe";

        using (Stream myStream = new FileStream(fName, FileMode.Create)) {
            bf.Serialize(myStream, data.image);
        }
    }
    catch (Exception) { }
}

Answer 1

I think you should avoid starting a new thread for each particular image. Since you have got just a single hard drive and store all files into the single directory, you should use just one disk writer thread. Then I'd recommend using some concurrent queue to transfer jobs from camera thread to disk writer thread. I don't show "code snippet" because this is not a thing you can write in good quality in a few lines of code.

Also you definitely must somewhere put 'new short[20000]' for each image, otherwise it is overwritten by next image before you save it to disk.

Also, I would expect that it is sufficient to write files in the main thread, because Windows uses concurrent techniques (mainly disk cache) automatically when you write data to disk. Are you sure that your hardware is fast enough to write all those data in real time?

Answer 2

When dealing with threads, ordering is no longer in your control. The thread pool can choose to schedule the threads in any order it likes. If you need things to happen sequentially in a specific order, threading does not make much sense anyway.

Regarding the corrupted images, it looks like the short[] image1 instance is being passed around. It is unclear what happens inside ReadImageFromCamera, but since you pass a pre-initialized array into it, chances are that the method will use that array and simply copy data into it (even though the ref keyword indicates that it might create a brand new array instance and assign that instead). Then you pass that array instance to WriteImageToFile on a separate thread.

Meanwhile, in parallell, you get the next image. Now you have a scenario where ReadImageFromCamera might write data into the array at the same time as WriteImageToFile is storing the data on disk. There you have your corrupted image. This can be avoided by passing a new array instance to WriteImageToFile:

ReadImageFromCamera(ref image1)
ImageData data;    

data.fileName = imageNumber;
data.image = (short[])image1.Clone(); // create a new array instance, so that
                                      // the next call to ReadImageFromCamera 
                                      // will not corrupt the data

ThreadPool.QueueUserWorkItem(WriteImageToFile, data); 

Still, as has been mentioned by Al Kepp, since you have only one hard drive, launching many threads might not be your best option here. You could look into having one long-running separate thread for storing data on disk, and putting the images into some sort of queue that the storage thread picks up data from and writes to disk. This comes with its own set of problems dealing with concurrency, limiting the size of the queue and what not.