How does the *NIX GUI work?

Question

I would like to know more about what makes the GUI on *NIX systems work, but not quite sure where to begin the research. From my understanding, the X server is what makes all the visuals possible, and atop that there are the various UI environments like KDE, Gnome and others.

But, for instance, I have always thought that running under certain UI environments restricted you to programs that utilize that environment, until I realized I'm able to use KDE utilities and software under Gnome and vice versa, which retain the look of their native environment.

What would be a good place to start learning about this topic?

Answer 1

I feel Paul's answer might be a bit jargon-heavy, so here's my attempt.

There are loads of display devices (VGA monitors, composite video, HDMI, etc.) and these tend to be handled directly in hardware, eg. in a dedicated graphics processor (GPU).

Rather than having our applications talk to this hardware directly, we use drivers (which live inside our operating system kernel). Different hardware needs different drivers, but all of the drivers can be given instructions using the same interface, for example OpenGL:

App --OpenGL--> Driver --> Hardware --VGA--> Screen

Of course, like most standards there are actually a whole bunch of different ones! OpenGL is supported by most drivers on most operating systems; its "OpenGL ES" sub-set works well on mobile phones and there are "software drivers" which can create images based on OpenGL instructions (and all drivers can draw images, although this is much slower than having real OpenGL support). A big competitor to OpenGL is DirectX, but that only works on Windows and XBox.

Rendering to OpenGL is fine for something like a full-screen 3D game, but the *NIX graphics system (known as "X") offers two major features on top of that: drawing multiple applications on the same screen and drawing over a network. To do this, a server process draws to the screen, and applications ("clients") talk to this server using the "X11 protocol" ("11" is just the version number):

App A ----------OpenGL-------+
                             |
App B --+                    |
        |                    |
        +--X11--> X server --+----> Driver --> Hardware --> Screen
        |
App C --+
        |
  ...network ...
        |
App D --+

X tends to have direct access to the drivers since it's been around longer than OpenGL, but that's not too important.

The X11 protocol works by having applications create windows which they're allowed to draw in. X can arrange these windows on the screen, including overlapping them. Applications using OpenGL can have their commands "pass through" X straight to the driver, and X will still arrange the window just like any other (that will not work over a network, since it's bypassing the networking abilities of X11).

We usually have an application dedicated to arranging, hiding/showing and closing windows for us, called a window manager. Optionally, a window manager might create some thin windows around the edges of the others, so that it can draw title bars, resizing handles, etc.

The X11 protocol includes commands for drawing shapes, rendering fonts, etc. and there are applications which use these directly, for example the xterm program and the twm window manager:

xterm --+
        |
        +--X11--> X --> Driver --> ...
        |
twm   --+

However, most modern applications find raw X11 too tedious; instead of drawing lines and shapes, we'd rather draw whole widgets (buttons, menus, icons, etc.). To do this, toolkits have been created. The two most famous are called Qt and GTK+ (the GIMP Tool Kit, since it was originally created for the GIMP); others include Motif, Lesstif, ETK, Tk and FLTK. We can ask the toolkit to draw a button and it will send all of the necessary X11 commands to draw a button, plus it will take care of the size and position, refreshing the drawing if something overlays it then moves away, telling our code when the button has been clicked, and some toolkits even allow changing what the widgets look like using themes. Some toolkits are also cross-platform, so they'll send X11 commands on Linux, different commands on Windows, OSX, etc.

Rhythmbox --> GTK+ --+
                     |
GIMP      --> GTK+ --+
                     |
Amarok    --> Qt   --+--X11--> X --> Driver --> ...
                     |
Skype     --> Qt   --+
                     |
aMSN      --> Tk   --+

Some toolkits offer features on top of others; for example wxWidgets gets Qt to do its drawing (on Linux; Windows and OSX are "native"), XUL is used by Firefox and uses GTK+ to do its drawing:

Audacity --> wxWidgets --> Qt   --+
                                  |
Firefox  --> XUL       --> GTK+ --+--X11--> X --> Driver --> ...
                                  |
GIMP     ----------------> GTK+ --+

An important point to mention is that the shape- and text-drawing commands of X11 aren't actually used very much, since they're very primitive. Many toolkits actually render their widgets as images, then get X to draw those images. The new Wayland system is trying to replace X by throwing away the drawing commands and letting applications and toolkits use OpenGL directly, which should make things much faster.

You mentioned different desktop environments, like GNOME and KDE, and whether they work together. Those are basically large collections of applications written to work well together. It just-so-happens that GNOME applications are all written with GTK+, whilst KDE applications are all written with Qt.

If you look at the arrows in my diagrams above, you'll notice that each Qt application talks to X separately, each GTK+ application talks to X separately, etc.; not only can a Qt and a GTK+ application work side-by-side, as far as X is concerned that's the same as two Qt applications or two GTK+ applications!

The only thing to worry about when mixing desktops is whether two applications are competing to do the same job, for example if you're trying to run two window managers or two desktop panels. Notice that this is not an issue of graphics, toolkits, etc. since I would get the same problems if I used two desktops built on the same toolkit (eg. lxpanel and gnome-panel are both written with GTK+ but they'll still get in each others' way!)