Multiple access to mmap objects in python

Question

I have a number of files, mapped to memory (as mmap objects). In course of their processing each file must be opened several times. It works fine, if there is only one thread. However, when I try to run the task in parallel, a problem arises: different threads cannot access the same file simultaneously. The problem is illustrated by this sample:

import mmap, threading

class MmapReading(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
    def run(self):
        for i in range(10000):
            content = mmap_object.read().decode('utf-8')
            mmap_object.seek(0)
            if not content:
                print('Error while reading mmap object')

with open('my_dummy_file.txt', 'w') as f:
    f.write('Hello world')
with open('my_dummy_file.txt', 'r') as f:
    mmap_object = mmap.mmap(f.fileno(), 0, prot = mmap.PROT_READ)

threads = []
for i in range(64):
    threads.append(MmapReading())
    threads[i].daemon = True
    threads[i].start()
for thread in threading.enumerate():
    if thread != threading.current_thread():
        thread.join()

print('Mmap reading testing done!')

Whenever I run this script, I get around 20 error messages.

Is there a way to circumvent this problem, other then making 64 copies of each file (which would consume too much memory in my case)?

Answer 1

The seek(0) is not always performed before another thread jumps in and performs a read().

1. Say thread 1 performs a read, reading to end of file; seek(0) has not yet been executed.
2. Then thread 2 executes a read. The file pointer in the mmap is still at the end of the file. read() therefore returns ''.
3. The error detection code is triggered because content is ''.

Instead of using read(), you can use slicing to achieve the same result. Replace:

    content = mmap_object.read().decode('utf-8')
    mmap_object.seek(0)

with

    content = mmap_object[:].decode('utf8')

content = mmap_object[:mmap_object.size()] also works.

Locking is another way, but it's unnecessary in this case. If you want to try it, you can use a global threading.Lock object and pass that to MmapReading when instantiating. Store the lock object in an instance variable self.lock. Then call self.lock.acquire() before reading/seeking, and self.lock.release() afterwards. You'll experience a very noticeable performance penalty doing this.

from threading import Lock

class MmapReading(threading.Thread):
    def __init__(self, lock):
        self.lock = lock
        threading.Thread.__init__(self)

    def run(self): 
        for i in range(10000):
            self.lock.acquire()
            mmap_object.seek(0)
            content = mmap_object.read().decode('utf-8')
            self.lock.release()
            if not content:
                print('Error while reading mmap object')

lock = Lock()
for i in range(64):
    threads.append(MmapReading(lock))
.
.
.

Note that I've changed the order of the read and the seek; it makes more sense to do the seek first, positioning the file pointer at the start of the file.