Does random.seed(seed) generate same sequence in multiple processes?

Question

Update on 7/19/2017

Found the solution here: https://stackoverflow.com/a/10021912/5729266

A quick conclusion if you don't want to read to the end.

The inconsistency of random numbers in my previous code was caused by thread-unsafe, because random module is considered as a global variable even each thread deals with one individual instance containing the random module.

To solve the problem, you have to use either thread-lock or generate an independent random instance as described in the link above. See test code below.

import threading
import random

class do_threads:

    def __init__(self):
        # Using random directly is thread-unsafe
        # self.random = random 

        # instead of using random, create a local random instance
        self.random = random.Random()

    def __call__(self, n):
        self.n = n
        self.run_thread()

    def get_balance(self, e):
        self.random.seed(self.n)
        return self.random.uniform(0, 1)


    def run_thread(self):
        total = []
        for i in range(100000):
           total.append(self.get_balance())
        print(sum(total) / 100000)

a = do_threads()
b = do_threads()

t1 = threading.Thread(target=a, args=(5,))
t2 = threading.Thread(target=b, args=(8,))
t1.start()
t2.start()
t1.join()
t2.join()

Old post:

In my Python program I need to run N subprocesses using multiprocessing.pool. Every subprocess spawns M threads, each of which needs to generate hashcode for IDs in column 'ID' of a dataframe.

The hash codes need to follow the distribution of uniform(0,1). To do this, I used ID as seed (random.seed(ID)) to set random state and then produced a random key from random.uniform(0, 1). But there were about 0.01% chance that an ID has different random numbers. For example, an ID ’200300’ appears 10000 times among all these threads/subprocesses, but 9999 times it has one random key and 1 time it has another random key.

So, my question is: Does random.seed(seed) generate same sequence in parallel programs all the time? If not, how can I fix the random state to ensure random.uniform(0, 1) to pop the same number given the same ID? I am also open for other methods that can hash ID into a random variable with uniform(0,1) distribution.

Just note, that I want to use Process and threads for my work and cannot concatenate these dataframes during the program to generate random keys all in once.

I tried using multiprocessing.Manager to share the random state or import random in parent process or pass random_generator() as instance or object from parent process to child environment. But it seems that things do not work as expected. 

Here is a simple version of my code:

#mythreads.py
from foo import Foo

class TaskWorker(Thread):
        def __init__(self, queue):
            Thread.__init__(self)
            self.queue = queue
        def run(self):
            while True:
                Foo, task = self.queue.get()
                Foo(task).generate_data(df)

def mythreads():
    queue = Queue()
    for x in range(10):
        worker = TaskWorker(queue)
        worker.daemon = True
        worker.start()
    for task in sub_list:
        queue.put((Foo, task))
    queue.join()

# foo.py
import random
class Foo:
    def __init__(self, task):
        ...

    def random_generator(self, e):
        random.seed(e)
        randomkey = random.uniform(0, 1)

    def generate_data(self, df):
        df['RK'] = df[‘ID’].apply(self.random_generator)
        ...


 
 


#main.py
from multiprocessing.pool import Pool
from mythreads import mythreads
with Pool(N) as p:
    p.map(mythreads, list_of_sublists)

Note: I use Python 3.6

Answer 1

Summary

Q. Does random.seed(seed) generate same sequence in parallel programs all the time?

A. Yes.

The random number generator is guaranteed reproduce the same series of random values given the same starting seed.

One other thought: Use random.random() instead of random.uniform(0, 1). Both give the same range of random variables, but the former is both faster and more idiomatic.

Example

Demonstration of separate processes running different generators starting with the same seed:

from multiprocessing.pool import Pool
from pprint import pprint
import random

def make_seq(identifier):
    random.seed(8675309)
    seq = [random.random() for i in range(4)]
    return identifier, seq

p = Pool(10)
pprint(list(p.map(make_seq, range(10))), width=100)

Output:

[(0, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (1, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (2, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (3, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (4, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (5, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (6, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (7, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (8, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507]),
 (9, [0.40224696110279223, 0.5102471779215914, 0.6637431122665531, 0.8607166923395507])]

Note that all the processes generated the same values.