In c# , how to iterate IEnumerable in multithreading environment

Question

I am in this situation that there is a large dictionary that is randomly updated by one thread at a fairly high frequency, and there is another thread that tries to take a snapshot of the dictionary to save as history. I currently using something like this:

Dictionary<string, object> dict = new Dictionary<string, object>();
var items = dict.Values.ToList();

This works fine for most of the time, except it occasionally throws:

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.

I understand why this happen, but I don't know what can I do to avoid the collection modified error.

What is the best approach to iterate such collection?

I also tried ConcurrentDictionary, but no luck. Why? Is ConcurrentDictionary thread safe only at item level?

Answer 1

According to the docs you should be able to use the GetEnumerator() method of ConcurrentDictionary to get a thread-safe iterator.

The enumerator returned from the dictionary is safe to use concurrently with reads and writes to the dictionary, however it does not represent a moment-in-time snapshot of the dictionary. The contents exposed through the enumerator may contain modifications made to the dictionary after GetEnumerator was called.

Since you're dealing with concurrent threads, it's not surprising to have some tradeoffs with consistency, but I would expect this approach to block less than the brute force approach given in other answers. This wouldn't have worked if you tried:

var items = concurrentDict.Items.ToList();

but it's supposed to work for

var items = concurrentDict.GetEnumerator();

or you could simply reference the iterator directly:

foreach(var item in concurrentDict)
{
    valueList.Add(item.Value);
}

Answer 2

An ImmutableDictionary might be appropriate for you, as it supports scalable multi-threading and snapshotting as part of its basic feature-set.

// initialize.
ImmutableDictionary<string, int> dict = ImmutableDictionary.Create<string,int>();

// create a new dictionary with "foo" key added.
ImmutableDictionary<string, int> newdict = dict.Add("foo", 0);

// replace dict, thread-safe, with a new dictionary with "bar" added.
// note this is using dict, not newdict, so there is no "foo" in it.
ImmutableInterlocked.TryAdd(ref dict, "bar", 1);

// take a snapshot, thread-safe.
ImmutableDictionary<string,int> snapshot = dict;

The immutable nature means that the dictionary can never change -- you can only add a value by creating a new dictionary. And because of this property, you take a "snapshot" of it by simply keeping a reference around from the point you want to snapshot.

It is optimized under the hood to be efficient, not copying the entire thing for every operation. That said, for other operations it isn't as efficient as ConcurrentDictionary, but it's all a trade-off in what you want. For instance, a ConcurrentDictionary can be concurrently enumerated but it's impossible to enumerate a snapshot of it.