PySide/PyQt - Starting a CPU intensive thread hangs the whole application

Question

I'm trying to do a fairly common thing in my PySide GUI application: I want to delegate some CPU-Intensive task to a background thread so that my GUI stays responsive and could even display a progress indicator as the computation goes.

Here is what I'm doing (I'm using PySide 1.1.1 on Python 2.7, Linux x86_64):

import sys
import time
from PySide.QtGui import QMainWindow, QPushButton, QApplication, QWidget
from PySide.QtCore import QThread, QObject, Signal, Slot

class Worker(QObject):
    done_signal = Signal()

    def __init__(self, parent = None):
        QObject.__init__(self, parent)

    @Slot()
    def do_stuff(self):
        print "[thread %x] computation started" % self.thread().currentThreadId()
        for i in range(30):
            # time.sleep(0.2)
            x = 1000000
            y = 100**x
        print "[thread %x] computation ended" % self.thread().currentThreadId()
        self.done_signal.emit()


class Example(QWidget):

    def __init__(self):
        super(Example, self).__init__()

        self.initUI()

        self.work_thread = QThread()
        self.worker = Worker()
        self.worker.moveToThread(self.work_thread)
        self.work_thread.started.connect(self.worker.do_stuff)
        self.worker.done_signal.connect(self.work_done)

    def initUI(self):

        self.btn = QPushButton('Do stuff', self)
        self.btn.resize(self.btn.sizeHint())
        self.btn.move(50, 50)       
        self.btn.clicked.connect(self.execute_thread)

        self.setGeometry(300, 300, 250, 150)
        self.setWindowTitle('Test')    
        self.show()


    def execute_thread(self):
        self.btn.setEnabled(False)
        self.btn.setText('Waiting...')
        self.work_thread.start()
        print "[main %x] started" % (self.thread().currentThreadId())

    def work_done(self):
        self.btn.setText('Do stuff')
        self.btn.setEnabled(True)
        self.work_thread.exit()
        print "[main %x] ended" % (self.thread().currentThreadId())


def main():

    app = QApplication(sys.argv)
    ex = Example()
    sys.exit(app.exec_())


if __name__ == '__main__':
    main()

The application displays a single window with a button. When the button is pressed, I expect it to disable itself while the computation is performed. Then, the button should be re-enabled.

What happens, instead, is that when I press the button the whole window freezes while the computation goes and then, when it's finished, I regain control of the application. The button never appears to be disabled. A funny thing I noticed is that if I replace the CPU intensive computation in do_stuff() with a simple time.sleep() the program behaves as expected.

I don't exactly know what's going on, but it appears that the second thread's priority is so high that it's actually preventing the GUI thread from ever being scheduled. If the second thread goes in BLOCKED state (as it happens with a sleep()), the GUI has actually the chance to run and updates the interface as expected. I tried to change the worker thread priority, but it looks like it can't be done on Linux.

Also, I try to print the thread IDs, but I'm not sure if I'm doing it correctly. If I am, the thread affinity seems to be correct.

I also tried the program with PyQt and the behavior is exactly the same, hence the tags and title. If I can make it run with PyQt4 instead of PySide I could switch my whole application to PyQt4

Answer 1

This is probably caused by the worker thread holding Python's GIL. In some Python implementations, only one Python thread can execute at a time. The GIL prevents other threads from executing Python code, and is released during function calls that don't need the GIL.

For example, the GIL is released during actual IO, since IO is handled by the operating system and not the Python interpreter.

Solutions:

1. Apparently, you can use time.sleep(0) in your worker thread to yield to other threads (according to this SO question). You will have to periodically call time.sleep(0) yourself, and the GUI thread will only run while the background thread is calling this function.

2. If the worker thread is sufficiently self-contained, you can put it into a completely separate process, and then communicate by sending pickled objects over pipes. In the foreground process, create a worker thread to do IO with the background process. Since the worker thread will be doing IO instead of CPU operations, it won't hold the GIL and this will give you a completely responsive GUI thread.

3. Some Python implementations (JPython and IronPython) do not have a GIL.

Threads in CPython are only really useful for multiplexing IO operations, not for putting CPU-intensive tasks in the background. For many applications, threading in the CPython implementation is fundamentally broken and it is likely to stay that way for the forseeable future.