Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why do I see extra line breaks when working with python multiprocessing pool?

Example:

from multiprocessing.dummy import Pool as ThreadPool

def testfunc(string):
    print string

def main():

    strings = ['one', 'two', 'three', ...]
    pool = ThreadPool(10)
    results = pool.map(testfunc, strings)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

This will not give us clear results with one result in one line:

one
two 
three

But mesh, that has random linebreaks, like

one 


two
three

four
five
...

Why does it happen? Can i output my data with one linebreak per function call?

P.S. Sometimes i have even no linebreaks or even spaces! P.P.S. Working under windows

like image 908
avasin Avatar asked May 31 '26 01:05

avasin


1 Answers

print is a non-atomic operation, so one print can be interrupted in the middle by another print in a different process. You can prevent two processes from calling print simultaneously by putting a Lock around it.

from multiprocessing.dummy import Pool as ThreadPool
from multiprocessing import Lock

print_lock = Lock()
def testfunc(string):
    print_lock.acquire()
    print string
    print_lock.release()

def main():

    strings = ['one', 'two', 'three', 'four', 'five']
    pool = ThreadPool(10)
    results = pool.map(testfunc, strings)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()
like image 152
Kevin Avatar answered Jun 01 '26 17:06

Kevin



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!