Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to retrieve multiple values returned of a function called through multiprocessing.Process

I have a scenario like this :

for each in content :
     pdf_output,job_id=createpdf(each)
     if pdf_output : 
        pdf_output = pdf_output + pdf_output

I am trying to parallelize the whole process .Something like this

 jobs=[]
    for each in content : 
       jobs.append(multiprocessing.Process(target=self.createpdf, args=(content)))

    for each in jobs :
         jobs.start()
    for each in jobs :
         jobs.join()

How do I sensibly do the task of

if pdf_output : 
            pdf_output = pdf_output + pdf_output

For each job ? How do I retrieve the 2 retun values sent by createpdf and work on it ? I think multiprocessing.Queue is a clue , but how do I implement this ?

like image 700
Nishant Avatar asked May 29 '12 10:05

Nishant


People also ask

What does multiprocessing map return?

It yields one result returned from the given target function called with one item from a given iterable. It is common to call map and iterate the results in a for-loop. The multiprocessing.

What is the difference between pool and process in multiprocessing?

Pool supports multiple tasks, whereas the multiprocessing. Process class supports a single task. The Pool is designed to submit and execute multiple tasks. For example, the map(), imap(), and starmap() functions are explicitly designed to perform multiple function calls in parallel.

How do you do multiprocessing in Python?

In this example, at first we import the Process class then initiate Process object with the display() function. Then process is started with start() method and then complete the process with the join() method. We can also pass arguments to the function using args keyword.

How do you return multiple values from a function?

Return multiple values by using passing arguments By Reference. Passing arguments By Reference is probably the most common way to return multiple values from a function. It uses the ByRef keyword to tell the compiler that the variable passed to the function is only a pointer to a memory location where the actual value of the variable is stored.

How to return multiple values from a class in Python?

Following are different ways 1) Using Object: This is similar to C/C++ and Java, we can create a class (in C, struct) to hold multiple values and return an object of the class. Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.

Why do we need multiprocessing?

These processes can also share a common database, or something like that to work together, but, many times, it will make more sense to use multiprocessing to do some processing, and then return results back to the main program. That's what we're going to cover here. Let's say we want to run a function over each item in an iterable. Let's just do:

Can a VBA function return more than one value?

VBA functions (and other programming languages as well) can only return one value. It's a common and useful practice that we need to return more values from a function.


1 Answers

You do not need queues for such a simple task. I would recommend to use pools. The Pool.map method can apply a function to a series of values in parallel:

import multiprocessing
def createpdf(data):
    return ("This is my pdf data: %s\n" % data, 0)


data = [ "My data", "includes", "strings and", "numbers like", 42, "and", 3.14]
number_of_processes = 5
results = multiprocessing.Pool(number_of_processes).map(createpdf, data)
outputs = [result[0] for result in results]
pdfoutput = "".join(outputs)
like image 54
brandizzi Avatar answered Oct 27 '22 00:10

brandizzi