Last year I read the excellent article “Hands-Free Coding” by Josh Comeau which went into detail about modifying his programming workflow to use dictation and eye-tracking after developing Cubital Tunnel Syndrome. I highly recommend reading the whole article and watching the demonstration videos of his approach.

This article was my first exposure to the dictation software Talon, which is specifically built to be hackable and naturally targets software developers.

If you know anything about me by now, it’s that I can’t use a computer without a tiling window manager anymore. I struggled so much when I switched to Windows that I ended up writing my own tiling window manager, komorebi.

After consuming the examples shared by Josh, I started thinking about how this approach could also be adapted to allow users suffering from repetitive-strain injuries to interact with a tiling window manager using dictaction software like Talon.

Earlier this week I recorded a brief proof-of-concept video showing how directional focusing, window movement and workspace cycling could be achieved using voice commands. Below, I’ll share the code that I wrote to achieve the functionality in that demo, and some additional notes, to hopefully help others that may one day need it.

Talon has a configuration home directory which varies depending on your operating system. On Windows, it will be under ~/AppData/Roaming/talon. Inside that configuration home, there should be a user directory (you can create this if there isn’t one there), in which you can add the following code snippets.

This first snippet is a Python script which defines a function called komorebic, which takes a single command argument. The command argument can be any valid komorebic.exe command, all of which can be found by running komorebic.exe --help.

An important thing to note here is that the komorebic.exe commands need to be executed using a subprocess method which allows for the shell=True argument. Without this, on Windows, each time the komorebic function is called, a new Command Prompt will visibly spawn.

# komorebi.py

import subprocess

from talon import Module

mod = Module()

@mod.action_class
class Actions:
    def komorebic(command: str):
        """Send a message to komorebi"""
        subprocess.check_output(f"komorebic.exe {command}", shell=True)

This second snippet is in Talon’s domain-specific language (DSL). On the left-hand side are the voice commands, and on the right hand side are the commands that will get executed.

All of the executed commands call our user.komorebic() function.

I’m sure that this can be made cleaner and DRY-er if you spend more time digging into the grammar of Talon’s DSL, but this is enough to get you started with core tiling window management functionality.

# komorebi.talon

focus left: user.komorebic("focus left")
focus right: user.komorebic("focus right")
focus up: user.komorebic("focus up")
focus down: user.komorebic("focus down")

move left: user.komorebic("move left")
move right: user.komorebic("move right")
move down: user.komorebic("move down")
move up: user.komorebic("move up")

next workspace: user.komorebic("cycle-workspace next")
previous workspace: user.komorebic("cycle-workspace previous")

toggle monocle: user.komorebic("toggle-monocle")
promote: user.komorebic("promote")

I believe that this approach can be replicated for other tiling window managers on other operating systems that expose a CLI which is able to send signals to the twm process such as bspwm on Linux, and yabai on macOS.

Although I do not have the hardware to try this out for myself, I think that a way to take this hands-free setup to the next level could be to enable focus-follows-mouse mode for komorebi and use eye-tracking as a mouse replacement.

In theory, this would allow you to set the window which has input focus just by looking at it.

Since komorebi also supports drag-to-swap (ie. you drag a window across the screen/to a different monitor and release the left mouse button when it’s on top of the window that you want to swap it with), I imagine this could also be used to rearrange windows across multiple monitors just by looking at them.

If you have any questions you can reach out to me on Mastodon and Twitter.

If you’re interested in what I read to come up with solutions like this one, you can subscribe to my Software Development RSS feed.

If you’d like to watch me writing code while explaining what I’m doing, you can also subscribe to my YouTube channel.

Finally, if you are a komorebi user and you have the means, please consider sponsoring the project. :)