How to integrate Python's GTK with gevent?

Question

Gtk is a GUI toolkit with bindings to Python. Gevent is a Python networking library built on top of libevent (libev on newer versions) and greenlets, allowing the usage of network functions inside greenlets without blocking the whole process.

Both Gtk an gevent have blocking main loops that dispatch events. How to integrate their main loops so that I can receive both network events and UI events on my application, without one blocking another?

The naive approach is to register an idle callback on Gtk's main loop, that is called instead whenever there is no Gtk event. In this callback, we yield the greenlet so network events can happen, also giving a small timeout, so the process does not busy waits:

from gi.repository import GLib
import gevent

def _idle():
    gevent.sleep(0.1)
    return True

GLib.idle_add(_idle)

This approach is far from ideal, because I have a delay of 100 miliseconds between UI events processing, and if I lower the value too much, I waste too much processor busy-waiting.

I want a better approach, where my process is truly asleep while there is no event to process.

PS: I've already found a Linux specific solution (that will probably work under MacOS, too). What I really need now is a working Windows solution.

Answer 1

Given current gevent API, I don't think there is a general solution, but I think there might be specific solutions for each platform.

Posix solution (tested under Linux)

Since GLib's main loop interface allow us to set the poll function, i.e. the function that takes a set of file descriptor and returns when one of them is ready, we define a poll function that relies on gevent's select to know when the file descriptors are ready.

Gevent does not exposes poll() interface, and select() interface is a little different, so we have to translate the arguments and the return value when calling gevent.select.select().

What complicate the matters a bit is that GLib does not exposes, through Python's interface, the specific function g_main_set_poll_func() that allows the trick. So we have to use the C function directly, for this, the ctypes module comes in handy.

import ctypes
from gi.repository import GLib
from gevent import select

# Python representation of C struct
class _GPollFD(ctypes.Structure):
    _fields_ = [("fd", ctypes.c_int),
                ("events", ctypes.c_short),
                ("revents", ctypes.c_short)]

# Poll function signature
_poll_func_builder = ctypes.CFUNCTYPE(None, ctypes.POINTER(_GPollFD), ctypes.c_uint, ctypes.c_int)

# Pool function
def _poll(ufds, nfsd, timeout):
    rlist = []
    wlist = []
    xlist = []

    for i in xrange(nfsd):
        wfd = ufds[i]
        if wfd.events & GLib.IOCondition.IN.real:
            rlist.append(wfd.fd)
        if wfd.events & GLib.IOCondition.OUT.real:
            wlist.append(wfd.fd)
        if wfd.events & (GLib.IOCondition.ERR.real | GLib.IOCondition.HUP.real):
            xlist.append(wfd.fd)

    if timeout < 0:
        timeout = None
    else:
        timeout = timeout / 1000.0

    (rlist, wlist, xlist) = select.select(rlist, wlist, xlist, timeout)

    for i in xrange(nfsd):
        wfd = ufds[i]
        wfd.revents = 0
        if wfd.fd in rlist:
            wfd.revents = GLib.IOCondition.IN.real
        if wfd.fd in wlist:
            wfd.revents |= GLib.IOCondition.OUT.real
        if wfd.fd in xlist:
            wfd.revents |= GLib.IOCondition.HUP.real
        ufds[i] = wfd

_poll_func = _poll_func_builder(_poll)

glib = ctypes.CDLL('libglib-2.0.so.0')
glib.g_main_context_set_poll_func(None, _poll_func)

I feel that there should be a better solution, because this way we need to know the specific version/name of GLib being used. This could be avoided if GLib exposed g_main_set_poll_func() in Python. Also, if gevent implements select(), it could well implement poll(), what would make this solution a lot more simple.

Windows partial solution (ugly and broken)

Posix solution fails on Windows because select() will only work with network sockets, what given Gtk handles aren't. So I thought on using GLib's own g_poll() implementation (what is a thin wrapper on Posix, it is a fairly complicated implementation on Windows) in another thread to wait for the UI events, and sync it with gevent's side in the main thread via a TCP socket. This is a very ugly approach, because it requires true threads (apart from greenlets that you probably would be using, if you are using gevent) and plain (non-gevent) sockets on the waiting thread side.

Too bad UI events on Windows are split by threads, so that one thread can not, by default, wait for events on another thread. The message queue on a specific thread is not created until you perform some UI stuff. So I had to create an empty WinAPI message box (MessageBoxA()) on the waiting thread (certainly there is a better way to do it), and mangle threads message queues with AttachThreadInput() so it can see the events of the main thread. All this via ctypes.

import ctypes
import ctypes.wintypes
import gevent
from gevent_patcher import orig_socket as socket
from gi.repository import GLib
from threading import Thread

_poll_args = None
_sock = None
_running = True

def _poll_thread(glib, addr, main_tid):
    global _poll_args

    # Needed to create a message queue on this thread:
    ctypes.windll.user32.MessageBoxA(None, ctypes.c_char_p('Ugly hack'),
                                     ctypes.c_char_p('Just click'), 0)

    this_tid = ctypes.wintypes.DWORD(ctypes.windll.kernel32.GetCurrentThreadId())
    w_true = ctypes.wintypes.BOOL(True)
    w_false = ctypes.wintypes.BOOL(False)

    sock = socket()
    sock.connect(addr)
    del addr

    try:
        while _running:
            sock.recv(1)
            ctypes.windll.user32.AttachThreadInput(main_tid, this_tid, w_true)
            glib.g_poll(*_poll_args)
            ctypes.windll.user32.AttachThreadInput(main_tid, this_tid, w_false)
            sock.send('a')
    except IOError:
        pass
    sock.close()

class _GPollFD(ctypes.Structure):
    _fields_ = [("fd", ctypes.c_int),
                ("events", ctypes.c_short),
                ("revents", ctypes.c_short)]

_poll_func_builder = ctypes.CFUNCTYPE(None, ctypes.POINTER(_GPollFD), ctypes.c_uint, ctypes.c_int)
def _poll(*args):
    global _poll_args
    _poll_args = args
    _sock.send('a')
    _sock.recv(1)

_poll_func = _poll_func_builder(_poll)

# Must be called before Gtk.main()
def register_poll():
    global _sock

    sock = gevent.socket.socket()
    sock.bind(('127.0.0.1', 0))
    addr = sock.getsockname()
    sock.listen(1)

    this_tid = ctypes.wintypes.DWORD(ctypes.windll.kernel32.GetCurrentThreadId())
    glib = ctypes.CDLL('libglib-2.0-0.dll')
    Thread(target=_poll_thread, args=(glib, addr, this_tid)).start()
    _sock, _ = sock.accept()
    sock.close()

    glib.g_main_context_set_poll_func(None, _poll_func)

# Must be called after Gtk.main()
def clean_poll():
    global _sock, _running
    _running = False
    _sock.close()
    del _sock

This far the application runs and reacts correctly to clicks and other user events, but nothing is drawn inside the windows (I can see the frame and the background buffer pasted into it). Some redraw command may be missing in the mangling of threads and messages queues. It is beyond my knowledge how to fix it. Any help? Any better idea on how to do it?

Answer 2

Or you can create a separate thread for handling the Gevent main loop. And you need a mechanism for switching safe from one thread to other.

1. to switch safe from gui thread to Gevent thread (something similar as Stackless already provided: switching between diff OS threads in Stackless Python). A solution to switch safe is to use libev.ev_async.

Example code:

def in_gevent_thread(data_to_send):
  #now you are in gevent thread, so you can use safe greenlet + and network stuff
    ...

def on_button_click():
  #now you are in gui thread
  safe_switch = gevent.core.async(gevent_hub.loop)
  safe_switch.callback = functools.partial(in_gevent_thread, data_to_send)
  safe_switch.send()

1. to switch back from gevent thread to gui thread should be easy. Something similiar to WinApi PostThreadMessage should be in gtk, too ...