What is the proper way to test a highly concurrent worker thread system?

Question

I am working on a system that consists of:

1. A database
2. A bunch of threads pumping rows into this database
3. A bunch of threads working on these rows

The crucial point is that no more than one worker should ever be operating on a row at the same time.

Of course there are a bunch of strategies here...I can lock things at the database level, I can use mutexes, etc.

But regardless of how I implement this, I need to be able to test the system to make sure I've done it right.

What is the proper way to test this?

All I have been doing is running hundreds of threads and continually checking for accidental overlap. The problem is, it's hit or miss. It's probability. Even if I run 500 threads for an hour, there could still be a thread that overlaps another, only rarely.

Also, how do I even properly check for overlaps? The "overlap checker" itself has a finite resolution that might be missing the actual overlaps that are happening...

I know concurrency is a complicated topic but surely there must be some best practices or recommended ways of testing such a system, besides just running it for a long time and crossing ones fingers...

Answer 1

You should just rely on the database, no need to cross the fingers: This is a core feature of any real database to support transactions / ACID (wikipedia). Do I'm missing something? What do you mean by 'threads could overlap' to work on the same row?