Why do we need locks for threads, if we have GIL?

Question

I believe it is a stupid question but I still can't find it. Actually it's better to separate it into two questions:

1) Am I right that we could have a lot of threads but because of GIL in one moment only one thread is executing?

2) If so, why do we still need locks? We use locks to avoid the case when two threads are trying to read/write some shared object, because of GIL twi threads can't be executed in one moment, can they?

Answer 1

GIL protects the Python interals. That means:

1. you don't have to worry about something in the interpreter going wrong because of multithreading
2. most things do not really run in parallel, because python code is executed sequentially due to GIL

But GIL does not protect your own code. For example, if you have this code:

self.some_number += 1 

That is going to read value of self.some_number, calculate some_number+1 and then write it back to self.some_number.

If you do that in two threads, the operations (read, add, write) of one thread and the other may be mixed, so that the result is wrong.

This could be the order of execution:

1. thread1 reads self.some_number (0)
2. thread2 reads self.some_number (0)
3. thread1 calculates some_number+1 (1)
4. thread2 calculates some_number+1 (1)
5. thread1 writes 1 to self.some_number
6. thread2 writes 1 to self.some_number

You use locks to enforce this order of execution:

1. thread1 reads self.some_number (0)
2. thread1 calculates some_number+1 (1)
3. thread1 writes 1 to self.some_number
4. thread2 reads self.some_number (1)
5. thread2 calculates some_number+1 (2)
6. thread2 writes 2 to self.some_number

EDIT: Let's complete this answer with some code which shows the explained behaviour:

import threading import time  total = 0 lock = threading.Lock()  def increment_n_times(n):     global total     for i in range(n):         total += 1  def safe_increment_n_times(n):     global total     for i in range(n):         lock.acquire()         total += 1         lock.release()  def increment_in_x_threads(x, func, n):     threads = [threading.Thread(target=func, args=(n,)) for i in range(x)]     global total     total = 0     begin = time.time()     for thread in threads:         thread.start()     for thread in threads:         thread.join()     print('finished in {}s.\ntotal: {}\nexpected: {}\ndifference: {} ({} %)'            .format(time.time()-begin, total, n*x, n*x-total, 100-total/n/x*100)) 

There are two functions which implement increment. One uses locks and the other does not.

Function increment_in_x_threads implements parallel execution of the incrementing function in many threads.

Now running this with a big enough number of threads makes it almost certain that an error will occur:

print('unsafe:') increment_in_x_threads(70, increment_n_times, 100000)  print('\nwith locks:') increment_in_x_threads(70, safe_increment_n_times, 100000) 

In my case, it printed:

unsafe: finished in 0.9840562343597412s. total: 4654584 expected: 7000000 difference: 2345416 (33.505942857142855 %)  with locks: finished in 20.564176082611084s. total: 7000000 expected: 7000000 difference: 0 (0.0 %) 

So without locks, there were many errors (33% of increments failed). On the other hand, with locks it was 20 times slower.

Of course, both numbers are blown up because I used 70 threads, but this shows the general idea.