Writing to a file in the background

Question

Given a method that writes to a text file

public void WriteToFile( ) {
    var file = "C:\\AsyncTest.txt";
    var writer = File.Exists( file ) ? File.AppendText( file ) : File.CreateText( file );
    writer.WriteLine( "A simulated entry" );
    writer.Close();
}

I need to simulate a scenario in which this method could be called in a loop, possibly dozens of times and must run asynchronously.

So I tried calling the method in a new thread like so (where writer is the class where WriteToFile lives)

//in a loop...
  Thread thread = new Thread( writer.WriteToFile );
  thread.Start(  );

Which works perfectly once, but throws an IO Exception that the file is being used by another process on subsequent iterations. Which makes perfect sense, actually, but I don't know how to work around it.

I tried using Join() like this

Thread thread = new Thread( writer.WriteToFile );
thread.Start(  );
thread.Join();

But that locks the calling thread until all the joined threads complete, which sort of defeats the purpose, no?

I tried using ThreadPool.QueueUserWorkItem(writer.WriteToFile);, but get the same IO exception.

I tried using a lock

private object locker = new object();

public void WriteToFile( ) {
  lock(locker){
    //same code as above
  }
}

But that had no apparent effect

I also tried using the Task class to no avail.

So how can I "stack up" these background threads to write to a single file without conflict, while not locking up the calling thread?

Answer 1

Another option is to create a queue. Have the main thread put strings on the queue and have a persistent background thread read the queue and write to the file. It's really easy to do.

private BlockingCollection<string> OutputQueue = new BlockingCollection<string>();

void SomeMethod()
{
    var outputTask = Task.Factory.StartNew(() => WriteOutput(outputFilename),
        TaskCreationOptions.LongRunning);

    OutputQueue.Add("A simulated entry");
    OutputQueue.Add("more stuff");

    // when the program is done,
    // set the queue as complete so the task can exit
    OutputQueue.CompleteAdding();

    // and wait for the task to finish
    outputTask.Wait();
}

void WriteOutput(string fname)
{
    using (var strm = File.AppendText(filename))
    {
        foreach (var s in OutputQueue.GetConsumingEnumerable())
        {
            strm.WriteLine(s);
            // if you want to make sure it's written to disk immediately,
            // call Flush. This will slow performance, however.
            strm.Flush();
        }
    }
}

The background thread does a non-busy wait on the output queue, so it's not using CPU resources except when it's actually outputting the data. And because other threads just have to put something on the queue, there's essentially no waiting.

See my blog, Simple Multithreading, Part 2 for a little bit more information.

Answer 2

You could use something like:

// To enqueue the write
ThreadPool.QueueUserWorkItem(WriteToFile, "A simulated entry");

// the lock
private static object writeLock = new object();

public static void WriteToFile( object msg ) {
    lock (writeLock) {
        var file = "C:\\AsyncTest.txt";

        // using (var writer = File.Exists( file ) ? File.AppendText( file ) : File.CreateText( file )) {
        // As written http://msdn.microsoft.com/it-it/library/system.io.file.appendtext(v=vs.80).aspx , File.AppendText will create the
        // file if it doesn't exist

        using (var writer = File.AppendText( file )) {
            writer.WriteLine( (string)msg );
        }
    }
}

And please, use using with files!