The X New Developer’s Guide: Xlib and XCB (2024)

Alan Coopersmith

The two most popular questions about Xlib and XCB may be"What are they?" and "What's the difference?"

Most programming languages make it awkward for Xapplications to spit raw X protocol down the network andtake apart the protocol coming back. Thus, X toolkits andapplications are a lot easier to write if some libraryhandles these jobs for them, providing an API that fits withthe programming language and environment for connecting tothe X server.

At the bottom level of the X client library stack are Xliband XCB, two helper libraries (really sets of libraries)that provide API for talking to the X server. Xlib and XCBhave different design goals, and were developed in differentperiods in the evolution of the X Window System.

Most application developers should call Xlib and XCBsparingly. Higher level toolkits provide more efficientprogramming models, and support features expected in modernapplications, including support for complexinternationalized input and output, accessibility, andintegration with desktop environments. However, sometimesapplications will find themselves needing to make calls tothe raw underlying X11 libraries for operations notsupported by toolkits. An application might need to makecalls to X extension API's not covered in the currentversion of the toolkit's programming model. It is alsocommon for drawing not to be wrapped by toolkit API's. [Yes?--po8]

The original C-language X11 API is libX11, often referred toas "Xlib". It was designed to look like a traditionallibrary API, hiding the fact that calls result in protocolrequests to a server. Calls that don't require a responsefrom the X server are queued in a buffer to be sent as abatch of requests to the server. Those that require aresponse flush all the buffered requests and then blockuntil the response is received.

Xlib's mix of synchronous and asynchronous behaviors causessome problems. Xlib's behaviour is often confusing to newprogrammers. Calls appear to work sometimes and not others,because it is not obvious which calls implicitly flush thebuffer. The asynchronous nature of many calls makes itdifficult to debug problems. When an error is reported, thestack trace shows the call that was being made when theerror was received and processed, often many calls after theone that caused the error. Finally, Xlib's synchronous callsincur avoidable round-trip latency. This latency has anotable effect on application performance; in particular,startup times are often greatly increased.

After many years of experience with Xlib, and learning fromit and other protocol interface libraries, a second attemptwas made at defining a C language binding for X11: the "X11C Binding" layer XCB. XCB makes the client-server nature ofthe protocol explicit in its design. The client is in chargeof deciding when to flush the request buffer, when to readresults and when to wait for the server to respond.

For instance, to lookup a window property, the Xlib code isa single function call:

XGetWindowProperty(dpy, win, atom, 0, 0, False, AnyPropertyType, &type_ret, &format_ret, &num_ret, &bytes_after, &prop_ret);

Xlib generates the request to the X server to retrieve theproperty and appends it to its buffer of requests. Sincethis is a request that requires a response, Xlib thenflushes the buffer, sending the contents to the Xserver. Next, Xlib waits until the X server processes allthe requests preceding the property retrieve request, andsends the property retrieve reply. Xlib then returns thereply to the client.

Xlib also provides convenience functions that wrap aproperty request. These convenience functions retrievespecific properties, knowing the details of each propertyand how to request and decode it. Examples includeXGetWMName and XGetWMHints. Some of these functions could bewritten outside Xlib, but many use Xlib internals innon-trivial ways and are thus inseparable. [Yes? --po8]

XCB on the other hand, provides functions generated directlyfrom the protocol descriptions in an "obvious" mechanisticway. XCB functions map directly onto the protocol, withseparate functions to put requests into the outgoing bufferand to read results back from the X server asynchronouslylater. The XCB version of the above code is:

prop_cookie = xcb_get_property (dpy, False, win, atom, XCB_GET_PROPERTY_TYPE_ANY, 0, 0);prop_reply = xcb_get_property_reply (dpy, prop_cookie, NULL);

The power of XCB is in allowing those two steps to have asmuch code as you want between them. The programmer decideswhen to wait for data, instead of being forced to wait forthe data returned by a request at the time the request isissued.

Example: Converting xwininfo from Xlib to XCB

The program xwininfo is a command-line utility to printinformation about windows on an X server. It knows inadvance, from the command line options, most of the data itneeds to request information for each window from theserver. Thus, xwininfo can make its requests all at once,and then wait for the results to start coming in. When usingthe -tree option to walk the window tree, xwininfo canrequest the data for all the children of the current windowat once, batching even further. On a local connection on asingle CPU server, this means less context switches betweenX client and server. On a multi-core/CPU server, the Xserver can process requests on one core while the client ishandling the responses on another core as they becomeavailable, improving performance. On remote connections, therequests can be grouped into packets closer to the MTU sizeof the connection, instead of just sending whatever requestsare in the buffer when a request is made that needs aresponse.

Version 1.1 of xwininfo was converted from Xlib to XCB byAlan Coopersmith. It was tested with a GNOME desktop sessionwith a few clients. xwininfo was run as "xwininfo -root-all": this started xwininfo at the root of the windowhierarchy and asked it to traverse the tree, requesting allthe information available for each window along the way. Inthis sample session it found 114 windows. (In X, a window issimply a container for drawing output and receivingevents. X windows are often regions of, or borders around,the "user windows"). When running locally on a four-coreIntel Nehalem CPU, both versions ran so fast (0.05 secondsor less) that the difference in time was too small toaccurately measure. To measure remote performance, "ssh -X"was used to tunnel an X11 connection from California to acomputer in China, and from there back to the workstation inCalifornia, introducing a huge amount of latency. With thissetup, the difference was dramatic between the two:

Xlib: 0.03u 0.02s 8:19.12 0.0% xcb: 0.00u 0.00s 0:45.26 0.0% 

Of course, xwininfo is an unusual X application in a few ways:

• xwininfo runs through the requests as fast as it can andthen exits, not waiting for user input (unless you usethe mode where you click on a window to choose it, afterwhich it runs through as normal). Once they are up andrunning, most X applications spend most of their timewaiting for user input, so the overall runtime won'tdecrease as much by reducing the time spentcommunicating with the X server. However, applicationstartups are typically dominated by round-trip times,and so proper use of XCB reduces the huge startup timesof X applications running over high-latency (and evenmedium-latency) connections.

• xwininfo uses only the core protocol and shapeextension. It does not use the more complex extensions,such as Render or Xinput, that most modern applicationsuse. Xinput, XKB, and GLX are especially problematic, asthose have not yet been fully supported in an XCBrelease, though support has been worked on through someGoogle Summer of Code projects.

• xwininfo is small enough that a complete reworking touse XCB in one shot was feasible. Most applications aremuch larger than this. XCB is aimed primarily at newcode and at toolkits: it is designed specifically tointeroperate with existing Xlib applications. Calls toXlib and XCB can be mixed, so Xlib applications can beconverted partially or incrementally if desired.

• xwininfo used only raw Xlib, without any toolkit. Thus,it did not have to worry about which X library thetoolkit used.

• xwininfo uses only a few of the Xlib helperfunctions. This makes it more directly mappable toXCB. Applications that rely on Xlib's input methodframework, compose key handling or character setconversion, for example, would be harder toport. Fortunately, modern toolkits handle most of thisfunctionality in the toolkit layer anyway.

xwininfo did rely on Xlib helper functions for convertingthe window name property from other character sets---the XCBversion currently only works for UTF-8 and Latin-1 windownames. Since most modern toolkits use UTF-8, no one islikely to notice. Older applications with localized windownames will fail, but there are few of these in use.

Mixing Xlib & XCB calls

As mentioned above, XCB provides a method for incrementalconversion from Xlib to XCB. One can use libX11 to open thedisplay and pass the Display pointer it returns to existingcode, toolkits, and libraries. To call an XCB function, onecan convert the Display pointer to an xcb_connection_tpointer for the same connection. This enables calling intoXlib and XCB from the same application.

Xlib and XCB compatibility was achieved by rebuilding libX11as a layer on top of libxcb. Xlib and XCB share the same Xserver connection and pass control of it back andforth. That option was introduced in libX11 1.2, and is nowalways present (no longer optional) since the 2010 releaseof libX11 1.4.

Example: Converting xdpyinfo extension queries to XCB

xdpyinfo is another command-line tool in the standard XWindow System toolset. Like xwininfo, xdpyinfo prints a lotof information about the X server. xdpyinfo calls manyextensions, and few of its calls block waiting for aresponse from the server. If you add the "-queryExt" option,though, for every extension xdpyinfo calls XQueryExtensionto print which request, event, and error ids are assigned tothat extension in the currently running server. These idsare dynamically assigned, and vary depending on the set ofextensions enabled in a given serverbuild/configuration. Thus, the list of extension ids iscritical information to have when debugging X error reportsthat reference them. Using "xdpyinfo -queryExt" isespecially needed when reporting an X error message thatcomes from a custom error handler like the one in the gtktoolkit: such error handlers typically omit the extensioninformation found in the default Xlib error handler, so theperson reading the bug report will be unable to identify theextension in which the error was encountered.

The Xlib call XQueryExtension takes one extension name at atime, sends a request to the X server for the id codes forthat extension, and waits for a response so it can returnthose ids to the caller. On the Xorg 1.7 server used as thetest system for this conversion, there were 30 active Xextensions, so that's 30 tiny packets sent to the X server,30 times that the xdpyinfo client blocks in poll() waitingfor a response, and 30 times that the X server goes throughthe client handling and request scheduling code before goingback to block again on its own select() loop.

Note: XListExtensions can be used to get a list of availableextensions that can be called with XQueryExtension.

A simple patch to xdpyinfo replaced just that loop of callsto XQueryExtension with two loops. The first loop calledxcb_query_extension for each extension. When the entirebatch of queries had been issued, a second loop calledxcb_query_extension_reply to start collecting the batchedreplies. Gathering system call counts with "truss -c" showedthe expected reduction in a number of system calls made bythe xdpyinfo client:

Over a TCP connection, the switch to XCB for thistransaction reduced both the number of packets and (due totcp packet header overhead) the overall amount of data:

This sort of change is far more feasible than wholesaleconversion to XCB for most applications. Find the hotspotswhere the application is waiting for data from the serverand convert those. There are almost always opportunities inapplication startup code, when the application is gatheringthe information about the X server and session. Convertingjust those calls to more efficient sets of XCB calls canhave major performance benefits. Earlier work by Xdevelopers reduced the latency of many applications byconverting repeated calls to XInternAtom with a single callto fetch multiple atoms at once via XInternAtoms. XCBpermits a generalization of this principle.

Extension libraries

Each new extension to the X11 protocol adds requests thatclients can make to the X server. To allow client softwareto utilize these requests, most extensions offer API's builton top of Xlib or XCB. These API's use the library'sconnection marshalling to include their requests in thestream sent to the X server.

In the early X11 releases, many of the smaller and morecommon extensions were grouped into a common library,libXext. You will find several there today which are stillin use, such as the MIT-SHM Shared Memory extension, theSHAPE extension for non-rectangular windows, and the SYNCextension for event synchronization. However, libXext alsoincludes some API's for extensions no longer found incurrent Xorg server releases, such as App-Group andLow-Bandwidth X (LBX), as well as extensions many apps neveruse, such as DPMS for display power management. Since theseextension API's cannot be removed from libXext withoutbreaking any existing application which may be using them,the code is stuck in there.

Accordingly, new Xlib extension API's are no longer added tolibXext. Instead a new library utilizing libX11 is createdfor each extension. Having a library per extension makes iteasier to evolve the API for that extension, to deprecate anobsolete extension and to only link it into the clients thatactually need it. Almost all modern extensions have theirown Xlib API library---libXrender for the RENDER extension,libXcomposite for the COMPOSITE extension, and so on. Ahandful of extensions are so core to the protocolinteraction that they are supported directly in libX11itself, such as BigRequests, XC-MISC, and XKB.

When XCB added its API style to the mix, it followed thenewer style and created a "libxcb"-prefixed library for eachextension---libxcb-composite, libxcb-render, etc. Since XCBcan generates the API code for an extension automaticallyfrom an XML description of the extension protocol, newextension API's are created by simply adding the extensiondescription to the xcb-proto package and rebuilding.Unfortunately, some of the older extensions have complexprotocols that are not easily described in the XMLsyntax. Work is ongoing to extend the syntax and codegenerator to handle these. The XKB & GLX protocols arecurrent challenges.

API documentation

To write code using Xlib or XCB, you'll need to know thedetails of the library API. Xlib includes man pages formost functions, providing a good API reference. libXextincludes man pages for some of the extension API's itincludes, but not all of them. Man page coverage is evenmore spotty in the individual Xlib-based extensionlibraries.

There's also a more complete guide to the Xlib API, andspecifications for many extension API's, in the X.Org onlinedoc set at http://www.x.org/releases/current/doc/.

For extensions without Xlib-style API documentation, thecalls are usually simple mappings to the protocolspecifications provided in the above-linked doc set.

For XCB, the documentation relies even more heavily on theprotocol specifications. The generated API is an exactmapping to the X protocol; it translates the C call datainto X protocol encoded packets as straightforwardly aspossible. The connection management and other functions inthe XCB API are documented athttp://xcb.freedesktop.org/XcbApi/. Work is in progress onadding support to XCB to generate Unix style reference manpages from the XML protocol descriptions as well, fordeveloper convenience.

There is also a XCB tutorial, "Basic Graphics ProgrammingWith The XCB Library" athttp://www.x.org/releases/current/doc/libxcb/tutorial/index.html.

Helping us improve our API documentation for either librarystack is always appreciated. See the Documentation chapterlater in this guide for more information.