Concurrent data structure design

Question

I am trying to come up with the best data structure for use in a high throughput C++ server. The data structure will be used to store anything from a few to several million objects, and no sorting is required (though a unique sort key can be provided very cheaply).

The requirements are that it can support efficient insert, ideally O(1), moderately efficient removal, and efficient traversal. It doesn't need to support a find operation (other than might be needed for removal).

The twist is that it must be thread safe with respect to modifications while other threads are enumerating the data structure. This means that a simple red-black tree doesn't work, since one thread can't insert an element (and perform the necessary tree rotations) without messing up any cursors held by other threads.

It is not acceptable to use a read/write lock and defer the write operations until all readers have finished, since read operations may be long lived. It doesn't matter if inserts that happen while there is a reader are visible to that reader or not.

Memory footprint is also very important, and small is obviously better!

What suggestions are there?

Response to comments:

Thanks for the answers.

No, inserts cannot invalidate existing iterators. Iterators may or may not see the new insert, but they must see everything that they would have seen if the insert hadn't occurred.

Deletion is required, however due to higher level rules I can guarantee that a iterator will never be stopped on an item that is available for deletion.

Locking per node for a cursor would have too great an impact on performance. There may be a number of threads reading at once, and any kind of memory hot spot that multiple threads are using in a lock kills memory bandwidth (as we discovered the hard way!). Even a simple count of readers with multiple threads calling InterlockedIncrement fails to scale cleanly.

I agree a linked list is likely the best approach. Deletes are rare, so paying the memory penalty for the back pointers to support O(1) delete is costly and we may compute those separately on demand and since deletes tend to be batch operations.

Fortunately insertion into a linked list doesn't require any locking for readers, as long as the pointers are updated in the inserted node before the head pointer is changed.

The lock-copy-unlock idea is interesting. The amount of data involved is too large for this to work as the default for readers, but it could be used for writers when they collide with readers. A read/write lock would protect the whole structure, and the write would clone the data structure if it collides with a reader. Writes are much rarer than reads.

Answer 1

Personally, I'm quite fond of persistent immutable data structures in highly-concurrent situations. I don't know of any specifically for C++, but Rich Hickey has created some excellent (and blisteringly fast) immutable data structures in Java for Clojure. Specifically: vector, hashtable and hashset. They aren't too hard to port, so you may want to consider one of those.

To elaborate a bit more, persistent immutable data structures really solve a lot of problems associated with concurrency. Because the data structure itself is immutable, there isn't a problem with multiple threads reading/iterating concurrently (so long as it is a const iterator). "Writing" can also be asynchronous because it's not really writing to the existing structure but rather creating a new version of that structure which includes the new element. This operation is made efficient (O(1) in all of Hickey's structures) by the fact that you aren't actually copying everything. Each new version shares most of its structure with the old version. This makes things more memory efficient, as well as dramatically improving performance over the simple copy-on-write technique.

With immutable data structures, the only time where you actually need to synchronize is in actually writing to a reference cell. Since memory access is atomic, even this can usually be lock-free. The only caveat here is you might lose data between threads (race conditions). The data structure will never be corrupted due to concurrency, but that doesn't mean that inconsistent results are impossible in situations where two threads create a new version of the structure based on a single old and attempt to write their results (one of them will "win" and the other's changes will be lost). To solve this problem, you either need to have a lock for "writing operations", or use some sort of STM. I like the second approach for ease-of-use and throughput in low-collision systems (writes are ideally non-blocking and reads never block), but either one will work.

You've asked a tough question, one for which there isn't really a good answer. Concurrency-safe data structures are hard to write, particularly when they need to be mutable. Completely lock-free architectures are provably impossible in the presence of shared state, so you may want to give up on that requirement. The best you can do is minimize the locking required, hence the immutable data structures.