Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to change multiprocessing shared array size?

I want to create a shared array with a dynamic size. I want to assign an array with an unknown size to it in another process.

from multiprocessing import Process, Value, Array


def f(a):
    b=[3,5,7]
    #resize(a,len(b)) # How to resize "a" ???
    a[:]=b  # Only works when "a" is initialized with the same size like the "b"


arr = Array('d', 0) #Array with a size of 0

p = Process(target=f, args=(arr))
p.start()
p.join()

print arr[:]
like image 577
Caniko Avatar asked Oct 25 '15 13:10

Caniko


People also ask

Does multiprocessing shared memory python?

Python provides the ability to create and manage new processes via the multiprocessing. Process class. In multiprocessing programming, we typically need to share data and program state between processes. This can be achieved using shared memory via shared ctypes.

Does torch Use multiprocessing?

torch. multiprocessing is a drop in replacement for Python's multiprocessing module. It supports the exact same operations, but extends it, so that all tensors sent through a multiprocessing. Queue , will have their data moved into shared memory and will only send a handle to another process.

What is shared memory python?

Shared memory can be a very efficient way of handling data in a program that uses concurrency. Python's mmap uses shared memory to efficiently share large amounts of data between multiple Python processes, threads, and tasks that are happening concurrently.

How do you do multiprocessing in Python?

In this example, at first we import the Process class then initiate Process object with the display() function. Then process is started with start() method and then complete the process with the join() method. We can also pass arguments to the function using args keyword.


1 Answers

The size of mp.Arrays can only be set once upon instantiation. You could use a mp.Manager to create a shared list however:

import multiprocessing as mp

def f(mlist):
    b = [3, 5, 7]
    mlist[:]=b  

if __name__ == '__main__':
    manager = mp.Manager()
    mlist = manager.list()
    p = mp.Process(target=f, args=[mlist])
    p.start()
    p.join()

    print(mlist[:])

yields

[3, 5, 7]

Note also args=(arr) results in

TypeError: f() takes exactly 1 argument (0 given)

because args expects a sequence of arguments to be passed to it. (args) evaluates to arr. To pass arr to f you would need args=[arr] or args=(arr,) (a tuple containing 1 element).

like image 125
unutbu Avatar answered Oct 24 '22 05:10

unutbu