Spawning a Subprocess With GLib and Gio

While I was working on my first GNOME application which I hope to publish soon I stumbled upon a small hindrance of how to spawn a subprocess efficiency without blocking the main thread. I could have left it as is with the straightforward pythonic way but I wanted the user interface to be fluid.

So how did I solved it? At first, I found a function in GLib that can spawn a subprocess called GLib.spawn_async, it was pretty simple to write to stdin and read from stdout which was necessary for what I wanted to do. When you spawn a process with the function you will get back a PID and file descriptors for stdin, stdout, and stderr. Writing to stdin was simply calling os.write with the given file descriptor and the bytes I wanted to write. Reading was also easy but a bit longer, I had to create a file object from the file descriptor using os.fdopen and read it like a normal file. The final code looked something like that:

import os
from gi.repository import GLib

pid, stdin_fd, stdout_fd, stderr_fd = GLib.spawn_async(cmd.split(), standard_input=True, standard_output=True, stndard_error=True)

os.write(stdin_fd, b'hello there')
os.close(stdin_fd)

with os.fdopen(stdout_fd) as stdout:
    print(stdout.read())

with os.fdopen(stderr_fd) as stderr:
    print(stderr.read())

GLib.spawn_close_pid(pid)

The problem with the code was that even though the process was spawned asynchronously, reading and writing is done on the main thread and will make the program unresponsive. Luckily for us, there’s a function called GLib.io_add_watch that can listen to a file descriptor and call a function when there’s new data. So to read stdout this way our code would look like that:

def callback(stdout_fd, cond):
    if cond != GLib.IO_IN:
	return False
	
    with os.fdopen(stdout_fd) as stdout:
	print(stdout.read())

    return True

GLib.io_add_watch(stdout_fd, GLib.IO_IN, callback)

Great, now we have a callback that will be called when we’ve got something to read from stdout but how do we cancel it? At that point, I decided to ask for help on IRC and I got a recommendation to use Gio.Subprocess instead of GLib.spawn_async. Indeed, Gio.Subprocess felt a lot more high level with many helpful functions.

So to create a new instance of the Gio.Subprocess class and spawn a new subprocess we can use the Gio.Subprocess.new constructor that only accept two arguments, the command and a set of flags to set the behaviour of the subprocess. To write to stdin and read from stdout, we will use the Gio.SubprocessFlags.STDIN_PIPE flag and the Gio.SubprocessFlags.STDOUT_PIPE flag.

Now that we have a subprocess object we have more than one way to communicate with it but since we want to do it asynchronously we will use the communicate_utf8_async method that will send the string that we give it to stdin and call a callback with the process output. The callback will receive the subprocess and the result but to get the actual output of the process (stdout and stderr) we need to call communicate_utf8_finish in the callback. This is the complete code for it:

from gi.repository import Gio

def callback(sucprocess: Gio.Subprocess, result: Gio.AsyncResult, data):
    _, stdout, stderr = proc.communicate_utf8_finish(result)
    print(stdout)
    print(stderr)

proc = Gio.Subprocess.new(cmd.split(), Gio.SubprocessFlags.STDIN_PIPE | Gio.SubprocessFlags.STDOUT_PIPE, Gio.SubprocessFlags.STDERR_PIPE)
proc.communicate_utf8_async('general kenobi', None, callback, None)

That’s it, you can test and see that even with a sleep in the subprocess your UI will continue to be responsive. But there was a reason I switched to Gio, I wanted to be able to cancel the callback. Notice that two arguments areNone in the call to communicate_utf8_async. The first one is of type Gio.Cancellable and the second is the user data to pass to the callback so of course, we care about the first None argument.

Creating an instance of Gio.Cancellable is easily done with the default constructor Gio.Cancellable.new does not have any required arguments. So we pass the cancellable object to the communicate_utf8_async method and if we want to cancel the operation we will just call the cancel method on Gio.Cancellable. Our previous code with the Gio.Cancellable object is this:

from gi.repository import Gio

def callback(sucprocess: Gio.Subprocess, result: Gio.AsyncResult, data):
    _, stdout, stderr = proc.communicate_utf8_finish(result)
    print(stdout)
    print(stderr)

proc = Gio.Subprocess.new(cmd.split(), Gio.SubprocessFlags.STDIN_PIPE | Gio.SubprocessFlags.STDOUT_PIPE, Gio.SubprocessFlags.STDERR_PIPE)
cancellable = Gio.Cancellable.new()
proc.communicate_utf8_async('general kenobi', cancellable, callback, None)
cancellable.cancel()

That code would cancel the task immediately but it will also raise an exception inside the callback because this is our way to know our callback was canceled. The exception that was raised is a generic GLib.Error so we need to inspect the error to be sure it’s a cancellation error. How do we do it? The cancellation error will have a domain property of g-io-error-quark and a code property of 19 so inside the except block just check for those values like so:

def callback(sucprocess: Gio.Subprocess, result: Gio.AsyncResult, data):
    try:
	_, stdout, stderr = proc.communicate_utf8_finish(result)
	print(stdout)
	print(stderr)
    except GLib.Error as err:
	if err.domain == 'g-io-error-quark' and err.code == GIO_CANCEL_ERROR_CODE:
	    return

That’s it, we have spawned a subprocess and received its output without blocking the main thread.