executing two class methods at the same time in Python

Question

I am sure many similar questions have been asked before, but after reading many of them I am still not very sure what I should do. So, I have a Python script to control some external instruments (a camera and a power meter). I have written class for both instruments by calling the C functions in the .dll files using ctypes. Right now it looks something like this:

for i in range(10):
    power_reading = newport.get_reading(N=100,interval=1) # take power meter reading
    img = camera.capture(N=10)
    value = image_processing(img) # analyze the img (ndarray) to get some values
    results.append([power_reading,value]) # add both results to a list

I want to start executing the first two lines at the same time. Both newport.get_reading and camera.capture take about 100ms-1s to run (they will run for the same time if I choose the correct arguments). I don't need them to start at EXACTLY the same time, but ideally the delay should be smaller than about 10-20% of the total run time (so less than 0.2s delay when take each take about 1s to run). From what I have read, I can use the multiprocessing module. So I try something like this based on this post:

def p_get_reading(newport,N,interval,return_dict):
    reading = newport.get_reading(N,interval,return_dict)
    return_dict['power_meter'] = reading

def p_capture(camera,N,return_dict):
    img = camera.capture(N)
    return_dict['image'] = img

for i in range(10):
    manager = multiprocessing.Manager()
    return_dict = manager.dict()
    p = multiprocessing.Process(target=p_capture, args=(camera,10))
    p.start()
    p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1))
    p2.start()
    p.join()
    p2.join()
    print(return_dict)

I have a few problems/questions:

1. I need to get the return values from both function calls. Using my current method, return_dict is only showing the entry for capture_img but not the power meter reading, why is that? It also read that I can use Queue, what is the best method for my current purpose?

2. How can I know whether both functions indeed start running at the same time? I am thinking of using the time module to record both the start and end time of both functions, maybe using some wrapper function to do the time logging, but will the use of multiprocessing pose any restrictions?

3. I usually run my code on an IDE (spyder), and from what I have read, I need to run in command prompt to see the output (I have some print statements inside the functions for debugging purposes). Can I still run in IDE for having both functions run at the same time?

Answer 1

Using a Lock may help with synchronisation:

import multiprocessing

def p_get_reading(newport, N, interval, lock, return_dict):
    lock.acquire()
    lock.release()
    reading = newport.get_reading(N, interval)
    return_dict['power_meter'] = reading

def p_capture(camera, N, lock, return_dict):
    lock.acquire()
    lock.release()
    img = camera.capture(N)
    return_dict['image'] = img

if __name__ == "__main__":    
    for i in range(10):
        manager = multiprocessing.Manager()
        return_dict = manager.dict()
        lock = multiprocessing.Lock()
        lock.acquire()
        p = multiprocessing.Process(target=p_capture, args=(camera,10,lock,return_dict))
        p.start()
        p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1,lock,return_dict))
        p2.start()
        lock.release()
        p.join()
        p2.join()
        print(return_dict)

The two Process objects can now be created and start()ed in any order as the main routine has already acquired the lock. Once released, the two processes will fight between themselves to acquire and release the lock, and be ready almost at the same time.

Also, note the use of if __name__ == "__main__" as this helps when multiprocessing makes new processes.

I must say this seems like an abuse of a Lock

Answer 2

An answer to your first question is simply no if you are doing in normal way, but yes if you want it to be. No because the target function cannot communicate back to spawning thread using a return. One way to do it is to use a queue and wrapper functions as following:

from threading import Thread 
from Queue import Queue 

def p_get_reading(newport,N,interval,return_dict):
   reading = newport.get_reading(N,interval,return_dict)
   return_dict.update({'power_meter': reading})
   return return_dict

def p_capture(camera,N,return_dict):
   img = camera.capture(N) 
   return_dict.update({'image': img})
   return return_dict

def wrapper1(func, arg1, arg2, queue):
   queue.put(func(arg1, arg2)) 

def wrapper2(func, arg1, arg2, arg3, queue):
    queue.put(func(arg1, arg2, arg3)) 

q = Queue()
Thread(target=wrapper1, args=(p_capture, camera, 10 , q)).start() 
Thread(target=wrapper2, args=(p_get_reading, newport, 100, 1, q)).start() 

Now q holds the updated and returned dict from p_capture() and p_get_reading().