Random numbers for multiple threads

Question

Problem

I intend to write a C++11 application for Linux which does some numerical simulation (not cryptography) based on approximately one million pseudorandom 32bit numbers. To speed things up, I'd like to perform the simulation in parallel threads using all cores of a desktop CPU. I'd like to use the Mersenne Twister mt19937 provided by boost as the PRNG, and I guess that for performance reasons I should have one such PRNG per thread. Now I'm unsure about how to seed them in order to avoid generating the same subsequence of random numbers in multiple threads.

Alternatives

Here are the alternatives that I have thought of so far:

1. Seed the PRNG for every thread independently from /dev/urandom.

   I'm a bit worried about the case when the system entropy pool gets exhausted, as I don't know how the system internal PRNG operates. Could it happen that I accidentially get consecutive seeds which exactly identify consecutive states of the Mersenne Twister, due to the fact that /dev/urandom is using a Mersenne Twister itself? Probably strongly related to my concerns for the next point.

2. Seed one PRNG from /dev/urandom and the others from that first one.

   Basically the same concern as well: is it good or bad to use one PRNG to seed another that uses the same algorithm? Or in other words, does reading 625 32bit integers from a mt19937 correspond directly to the internal state of the mt19937 generator at any point during this generation?

3. Seed others from first with non-Mersenne information.

   As using the same algorithm to generate random numbers and to generate the initial seed feels somehow like it might be a bad idea, I thought about introducing some element which is not dependent on the Mersenne Twister algorithm. For example, I could XOR the thread id into each element of the initial seed vector. Does that make things any better?

4. Share one PRNG among threads.

   This would make sure that there is only one sequence, with all the known and desirable properties of the Mersenne Twister. But the locking overhead required to control access to that generator does worry me somewhat. As I have found no evidence to the contrary, I assume that I as the library user would be responsible for preventing concurrent access to the PRNG.

5. Pre-generate all random numbers.

   This would have one thread generate all the required 1M random numbers up front, to be used by the different threads later on. The memory requirement of 4M would be small compared to that of the overall application. What worries me most in this approach is that the generation of random numbers itself is not concurrent. This whole approach also doesn't scale too well.

Questions

Which of these approaches would you suggest, and why? Or do you have a different suggestion?

Do you know which of my concerns are justified and which are simply due to my lack of insight into how things actually work?

Answer 1

I would go with #1, seed every prng from urandom. This ensures that the states are totally independent (as far as the seed data is independent). Typically there will be plenty of entropy available unless you have many threads. Also, depending on the algorithm used for /dev/urandom you almost certainly don't need to worry about it.

So you might use something like the following to create each prng:

#include <random>  std::mt19937 get_prng() {     std::random_device r;     std::seed_seq seed{r(), r(), r(), r(), r(), r(), r(), r()};     return std::mt19937(seed); } 

You should verify that your implementation of std::random_device pulls from /dev/urandom under your configuration. And if it does use /dev/urandom by default then usually you can say std::random_device("/dev/random") if you want to use /dev/random instead.

Answer 2

You could use a PRNG with a different algebraic structure to seed the different PRNGs. E.g. some sequence of MD5 hashes.

However I would opt for #5. If it works then it is fine. If it does not you can still optimize it.

The point is creating a good PRNG is much harder than one might expect. A good PRNG for threaded applications is most probably something that is still subject to research.

If the number of CPUs is low enough you could get away with leap frogging. E.g. if you have 4 cores, initialize all with the same values, but advance core 1 PRNG by 1, #2 by and #3 by 3. Then advance always 4 steps when you need a new number.

Answer 3

I'd use one instance to seed the others. I'm pretty sure you can do this safely fairly easily.

• Even small changes in the state space cause fairly large changes downstream - if you can ensure they don't have exactly the same starting space (and no identical state prefix), I wouldn't worry about producing identical numbers. For instance, using just the values 1,2,3 to seed three threads would work fine - you don't even need to seed the entire space. Another advantage: by using clearly predictable seeds you can easily discredit the idea that you're cherry-picking any runs (assuming you're trying to demonstrate something).
• It's trivial to seed in a way that means the resultant "children" are highly un-correlated. Just iterate in a breadth-first fashion; i.e. if you want to seed N x 623 int values, don't seed 623 values sequentially, but pick the first N and distribute, then the next N etc. Even if there's some correlation between the seeder and the children, the correlation between the various children should be virtually non-existant - and that's all you care about.
• I'd prefer an algorithm that allows deterministic execution whenever possible, so depending on urandom is not attractive. This makes debugging easier.
• Finally, and obviously - test. These PRNG are fairly robust, but by all means eyeball the results and do a few correlation tests inspired by what you're simulating. Most problems should be obvious - either you've seeded badly and there are obvious repeating subsequences, you you've seeded well, and then the quality is dictated by the PRNG limitations.
• For final executions, after you're done testing, you can seed the first of 623 state values using urandom for peace of mind and/or the thread ID.