PyQt5 and asyncio: yield from never finishes

Question

I'm trying to create a new application based on PyQt5 and asyncio (with python 3.4, looking forward to eventually upgrade to 3.5 with async/await). My goal is to use asyncio so that the GUI stays responsive even when the application is waiting for some connected hardware to finish an operation.

When looking how to merge the event loops of Qt5 and asyncio, I found a mailing list posting, suggesting to use quamash. However, when running this example (unmodified), the

yield from fut

nevers seems to return. I see the output 'Timeout', so the timer callback obviously fires, but the Future fails to wake up the waiting method. When manually closing the window, it tells me that there are uncompleted futures:

Yielding until signal...
Timeout
Traceback (most recent call last):
  File "pyqt_asyncio_list.py", line 26, in <module>
    loop.run_until_complete(_go())
  File "/usr/local/lib/python3.5/site-packages/quamash/__init__.py", line 291, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

I tested this on Ubuntu with python 3.5 and on Windows with 3.4, same behaviour on both platforms.

Anyway, since this is not what I actually try to achieve, I tested some other code as well:

import quamash
import asyncio
from PyQt5.QtWidgets import *
from PyQt5.QtCore import *

@asyncio.coroutine
def op():
  print('op()')

@asyncio.coroutine
def slow_operation():
  print('clicked')
  yield from op()
  print('op done')
  yield from asyncio.sleep(0.1)
  print('timeout expired')
  yield from asyncio.sleep(2)
  print('second timeout expired')

def coroCallHelper(coro):
  asyncio.ensure_future(coro(), loop=loop)

class Example(QWidget):

  def __init__(self):
    super().__init__()
    self.initUI()

  def initUI(self):
    def btnCallback(obj):
      #~ loop.call_soon(coroCallHelper, slow_operation)
      asyncio.ensure_future(slow_operation(), loop=loop)
      print('btnCallback returns...')

    btn = QPushButton('Button', self)
    btn.resize(btn.sizeHint())
    btn.move(50, 50)
    btn.clicked.connect(btnCallback)

    self.setGeometry(300, 300, 300, 200)
    self.setWindowTitle('Async')    
    self.show()

with quamash.QEventLoop(app=QApplication([])) as loop:
  w = Example()
  loop.run_forever()
#~ loop = asyncio.get_event_loop()
#~ loop.run_until_complete(slow_operation())

The program is supposed to display a window with a button in it (which it does), with the button invoking slow_operation() without blocking the GUI. When running this example, I can click the button as often as I want, so the GUI is not blocked. But the

yield from asyncio.sleep(0.1)

is never passed and the terminal output looks like this:

btnCallback returns...
clicked
op()
op done
btnCallback returns...
clicked
op()
op done

There is no exception thrown when I close the window this time. The slow_operation() function basically works if I directly run the event loop with it:

#~ with quamash.QEventLoop(app=QApplication([])) as loop:
  #~ w = Example()
  #~ loop.run_forever()
loop = asyncio.get_event_loop()
loop.run_until_complete(slow_operation())

Now, two questions:

1. Is this a sensible way to achieve decoupling of lengthy operations from the GUI, generally? My intention is that the button callback posts the coroutine call to the event loop (with or without an additional level of nesting, cf. coroCallHelper()), where it is then scheduled and executed. I don't need separate threads, as it is really only I/O that takes time, no actual processing.

2. How can I fix this behaviour?

Thanks, Philipp

Answer 1

Ok, that's one plus of SO: Writing down a question makes you think again about everything. Somehow I just figured it out:

Looking again at the example from the quamash repo, I found that the event loop to use is obtained somewhat differently:

app = QApplication(sys.argv)
loop = QEventLoop(app)
asyncio.set_event_loop(loop)  # NEW must set the event loop

# ...

with loop:
    loop.run_until_complete(master())

The key seems to be the asyncio.set_event_loop(). It is also important to note that the QEventLoop mentioned there is the one from the quamash package, NOT from Qt5. So my example now looks like this:

import sys
import quamash
import asyncio
from PyQt5.QtWidgets import *
from PyQt5.QtCore import *

@asyncio.coroutine
def op():
  print('op()')


@asyncio.coroutine
def slow_operation():
  print('clicked')
  yield from op()
  print('op done')
  yield from asyncio.sleep(0.1)
  print('timeout expired')
  yield from asyncio.sleep(2)
  print('second timeout expired')

  loop.stop()

def coroCallHelper(coro):
  asyncio.ensure_future(coro(), loop=loop)

class Example(QWidget):

  def __init__(self):
    super().__init__()
    self.initUI()

  def initUI(self):
    def btnCallback(obj):
      #~ loop.call_soon(coroCallHelper, slow_operation)
      asyncio.ensure_future(slow_operation(), loop=loop)
      print('btnCallback returns...')

    btn = QPushButton('Button', self)
    btn.resize(btn.sizeHint())
    btn.move(50, 50)
    btn.clicked.connect(btnCallback)

    self.setGeometry(300, 300, 300, 200)
    self.setWindowTitle('Async')    
    self.show()

app = QApplication(sys.argv)
loop = quamash.QEventLoop(app)
asyncio.set_event_loop(loop)  # NEW must set the event loop

with loop:
    w = Example()
    w.show()
    loop.run_forever()
print('Coroutine has ended')

And it 'just works' now:

btnCallback returns...
clicked
op()
op done
timeout expired
second timeout expired
Coroutine has ended

Maybe this is of some help for others. I'm happy with it at least ;) Comments on the general pattern are still welcome, of course!

Addendum: Please note that this works with recent Python versions up to Python 3.7.x if quamash is replaced by asyncqt. However, using the same code with Python 3.8 causes the @coroutine decorators to generate RuntimeWarnings and eventually fails with a RuntimeError: no running event loop in asyncio.sleep(). Maybe someone else knows what to change to get this working again. It might just be that asyncqt is not yet compatible with Python 3.8.

Regards, Philipp