Menu
I'm using tensorflow to preprocess some large images. I was having a problem where the memory was rapidly collapsing. I turned to use multiprocessing in python so the memory would free up entirely whenever I want.
The thing is, I'm using python's multiprocess queues and for some reason unknown I can't pass my tensorflow session from my parent process to the children. Using some advanced debugging techniques (i.e. printing something every few lines) I noticed that python just goes idle inside the line where I make use of the session, it doesn't throw an error message.
My code looks something like this:
def subprocess(some_image, sess, q):
with sess.as_default():
# ... use sess and q ...
print "All good and well" #This is printed
some_image.eval() #Nothing happens here in console
print "Still all good and well" #This is not printed
if __name__ == '__main__':
# ... some initial operations ...
some_image = read_some_image()
sess = tf.Session()
q = Queue()
q.put(something)
p = Process(target=subprocess, args=(some_image, sess, q))
p.start()
p.join()
What could be the problem? Many thanks!
745
The multiprocessing package supports spawning processes. It refers to a function that loads and executes a new child processes. For the child to terminate or to continue executing concurrent computing,then the current process hasto wait using an API, which is similar to threading module.
Multiprocessor system facilitates parallel program execution and read/write sharing of data and thus may cause the processors to concurrently access location in the shared memory. Therefore, a correct and reliable mechanism is needed to serialize this access.
In multiprocessing , multiple Python processes are created and used to execute a function instead of multiple threads, bypassing the Global Interpreter Lock (GIL) that can significantly slow down threaded Python programs.
multiprocessing provides two methods of doing this: one using shared memory (suitable for simple values, arrays, or ctypes) or a Manager proxy, where one process holds the memory and a manager arbitrates access to it from other processes (even over a network).
I don't think you can share "state" as in the tf.Session() between processes like that.
I would think that each process needed it's own session.
109
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
