Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Object properties does not seem change in Parallel processing using joblib in python

I have the following code that works well with one process but doesn't work with more than one. No error messages but class properties doesn't not seem to save and I have no clue why or how to fix it. I am running the code under windows

class T:
  a = 0
  b = 0
  c = 0
  def do_something(self):
    self.a = 10
    self.b = 5
    self.c = 1
    return 'ok'

def call_T(a):
  return a.do_something()

if __name__ == '__main__':
  B = T()
  print(B.a)
  B.do_something()
  print(B.a)
  C = [T() for i in range(20)]
  print(C[14].c)
  F = Parallel(n_jobs=2)(delayed(call_T)(C[i]) for i in range(20))
  print(F)
  print(C[14].b)

results are

0
10
0
['ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok']
0

now for the same code only change n_jobs=1 and it will give me the correct expected resuls

0
10
0
['ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok']
5

I am clueless as what is wrong. Any help is appreciated

like image 336
mavios Avatar asked Aug 18 '26 03:08

mavios


1 Answers

I found a simple answer it turns out that joblib only passes results from returned method but not the memory using threading, however will pass both

class T:
  a = 0
  b = 0
  c = 0
  def do_something(self):
    self.a = 10
    self.b = 5
    self.c = 1
    return 'ok'

def call_T(a):
  return a.do_something()

if __name__ == '__main__':
  B = T()
  print(B.a)
  B.do_something()
  print(B.a)
  C = [T() for i in range(20)]
  print(C[14].c)
  F = Parallel(n_jobs=3,backend="threading")(delayed(call_T)(C[i]) for i in range(20))
  print(F)
  print(C[14].b)

results

0
10
0
['ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok', 'ok']
5
like image 91
mavios Avatar answered Aug 20 '26 17:08

mavios



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!