Why dwm’s Window Swallowing Patch Can’t Swallow tmux
Author: Jake Bauer | Published: 2020-06-26
Note: Code snippets in this blog post are governed by the MIT/X Consortium License as they are snippets taken from dwm code. See the dwm LICENSE file.
The dwm swallow patch is my favourite patch for dwm because it allows me to
launch GUI applications from my terminal and have the launched application take
over the terminal instead of spawning next to the terminal and rendering the
terminal window useless. You can sort of do this by just appending & and then
closing the terminal, but you also don’t get the terminal back once the
application has closed. Something like launching a video with mpv <video> and
having the mpv window replace the terminal window is very nice.
I was also playing around with running every terminal window with tmux specifically because it would allow me to remove the scrollback patch from my terminal emulator st and becuse it handles redrawing the text when the terminal window is resized. However, I noticed that launching applications from within a terminal running tmux wouldn’t get swallowed. A brief search on the interwebs turned up no useful information, so I decided to get my hands dirty and investigate why this was the case.
Before we get started: yes, I know I should learn to use gdb. I have used it before but frequently find printfs do a good enough job that I don’t have to faff about with gdb.
Jump down to the summary to skip the journey and go right to the explanation.
Demonstration
Take a look at the animations below to see what happens when a GUI application is launched while tmux is running:
Click for a higher resolution version of the above video.
Click for a higher resolution version of the above video.
Beginning the Search
So, what’s the difference between a normal terminal launching a GUI application and a terminal running tmux launching a GUI application?
By this point, I had no idea if the actual cause was something wrong with my
configuration of dwm. Perhaps my recent conversion from the scratchpad to the
named-scratchpads patch messed something up so I started by looking at the
values of the rules[] array in config.h to see if my terminal window was
being wrongly interpreted as being unable to swallow:
static const Rule rules[] = {
/* class instance title tags mask isfloating isterminal noswallow monitor scratch key*/
{ "Galculator", NULL, NULL, 0, 1, 0, -1, -1, 0 },
{ "Gimp", NULL, NULL, 0, 0, 0, -1, -1, 0 },
{ "Firefox", NULL, NULL, 1 << 1, 0, 0, -1, -1, 0 },
{ "St", NULL, NULL, 0, 0, 1, 0, -1, 0 },
{ NULL, NULL, "scratchpad", 0, 1, 1, 1, -1, 's' },
{ NULL, NULL, "calculator", 0, 1, 1, 1, -1, 'c' },
{ NULL, NULL, "Event Tester", 0, 1, 0, 1, -1, 0 },
};
These rules are interpreted by the function applyrules() which applies rules
to a launched client based on what’s in the rules[] array. A few printfs
later and I had found out that everything was being correctly interpreted here.
Finding The Answer
Since that thread didn’t lead anywhere, I examined the content of the swallow
patch itself to see what code it added and where I should look next. It led me
to look inside the manage() function which calls swallow() at the very end
if the variable term is a truthy value.
void
manage(Window w, XWindowAttributes *wa)
{
...
if (term)
swallow(term, c);
focus(NULL);
}
term is of type Client * and is set to NULL at the beginning of the
function:
void
manage(Window w, XWindowAttributes *wa)
{
Client *c, *t = NULL, *term = NULL;
...
}
term is then later set by the termforwin() function:
void
manage(Window w, XWindowAttributes *wa)
{
...
if (XGetTransientForHint(dpy, w, &trans) && (t = wintoclient(trans))) {
c->mon = t->mon;
c->tags = t->tags;
} else {
c->mon = selmon;
applyrules(c);
term = termforwin(c);
}
...
}
termforwin() returns a Client * value which represents the terminal which
launched the client application. It determines which terminal window the GUI
application was just launched from so dwm knows which window should be
swallowed.
My printfs up to this point told me that termforwin() was returning NULL
when a GUI application was launched inside of a terminal running tmux which is
why swallow() was not being called for these applications:
Applying rules to client st
Rules for window:
Title: (null)
Class: St
Instance: (null)
IsTerminal: 1
NoSwallow: 0
IsFloating: 0
Tags: 0
Scratchkey:
IN MANAGE(): term: (nil)
Applying rules to client org.pwmt.zathura
IN MANAGE(): term: 0x5626932ad110 <--- No tmux
Entered swallow function
IN SWALLOW(): org.pwmt.zathura should be swallowing st
Applying rules to client org.pwmt.zathura
IN MANAGE(): term: (nil) <--- With tmux
Now I had to take a look inside termforwin() to discover why it was returning
NULL for these windows. Since there were only two places where NULL could be
returned, I checked which of the two conditions were failing. It turned out that
the second return NULL; was being reached which meant that the if statement
inside the for loops was evaluating to false for every value of c.
Client *
termforwin(const Client *w)
{
Client *c;
Monitor *m;
if (!w->pid || w->isterminal)
return NULL;
for (m = mons; m; m = m->next) {
for (c = m->clients; c; c = c->next) {
if (c->isterminal && !c->swallowing && c->pid && isdescprocess(c->pid, w->pid))
return c;
}
}
return NULL;
}
A few more printfs later and I discovered that the cause of the failure was
that isdescprocess() was returning 0. isdescprocess() is a function that
determines if a process is a descendant of another process by walking up the
process tree. In the case above, it is trying to determine if the next client
c in the list of clients is a parent process of our newly launched process
w.
This check consistently failed every time a GUI application was launched from a
terminal running tmux which meant that the variable term was always NULL. To
get a closer look at why this might be failing, I added some more printfs in
strategic locations and ran pstree -g alongside every test I performed.
Here are the three scenarios I tested with relevant sections from the output log
I created (dwm.log.6) along with the relevant snippets from the pstree
output:
Zathura with Just st
MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client st
TERMFORWIN(): pid: 33340, isterminal: 1
MANAGE(): term: (nil)
MANAGE(): if term exists then c will swallow
MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client st
TERMFORWIN(): pid: 33369, isterminal: 1
MANAGE(): term: (nil)
MANAGE(): if term exists then c will swallow
MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client org.pwmt.zathura
TERMFORWIN(): pid: 33382, isterminal: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33369, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33340, isdescproc: 33340
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
MANAGE(): term: 0x55b88574e0e0
MANAGE(): if term exists then c will swallow
MANAGE(): swallowing...
IN SWALLOW()
SWALLOW(): org.pwmt.zathura should be swallowing st
=======================================================
systemd(1)-+-ModemManager(1278)-+-{ModemManager}(1278)
|-login(33254)---startx(33254)---xinit(33254)-+-Xorg(33293)-+-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | `-{Xorg}(33293)
| `-dwm(33305)-+-compton(33305)-+-{compton}(33305)
| | |-{compton}(33305)
| | |-{compton}(33305)
| | `-{compton}(33305)
| |-dunst(33305)-+-{dunst}(33305)
| | `-{dunst}(33305)
| |-lxpolkit(33305)-+-{lxpolkit}(33305)
| | `-{lxpolkit}(33305)
| |-slstatus(33305)
| |-st(33340)---bash(33341)---zathura(33382)-+-{zathura}(33382)
| | |-{zathura}(33382)
| | |-{zathura}(33382)
| | |-{zathura}(33382)
| | |-{zathura}(33382)
| | |-{zathura}(33382)
| | `-{zathura}(33382)
| |-st(33369)---bash(33370)---pstree(33390)
| |-unclutter(33305)
| `-xautolock(33305)
Zathura with st Running tmux
MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client org.pwmt.zathura
TERMFORWIN(): pid: 33414, isterminal: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33369, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33340, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
TERMFORWIN(): Passed pid and isterminal check but returning NULL
MANAGE(): term: (nil)
MANAGE(): if term exists then c will swallow
===========================================================
systemd(1)-+-ModemManager(1278)-+-{ModemManager}(1278)
|-login(33254)---startx(33254)---xinit(33254)-+-Xorg(33293)-+-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | `-{Xorg}(33293)
| `-dwm(33305)-+-compton(33305)-+-{compton}(33305)
| | |-{compton}(33305)
| | |-{compton}(33305)
| | `-{compton}(33305)
| |-dunst(33305)-+-{dunst}(33305)
| | `-{dunst}(33305)
| |-lxpolkit(33305)-+-{lxpolkit}(33305)
| | `-{lxpolkit}(33305)
| |-slstatus(33305)
| |-st(33340)---bash(33341)---tmux: client(33397)
| |-st(33369)---bash(33370)---pstree(33418)
| |-unclutter(33305)
| `-xautolock(33305)
|-tmux: server(22354)-+-bash(22355)
| |-bash(22513)
| |-bash(24710)
| `-bash(33398)---zathura(33414)-+-{zathura}(33414)
| |-{zathura}(33414)
| `-{zathura}(33414)
Zathura with st Running screen
APPLYRULES(): applying to client org.pwmt.zathura
TERMFORWIN(): pid: 33439, isterminal: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33369, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33340, isdescproc: 33340
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
MANAGE(): term: 0x55b88574e0e0
MANAGE(): if term exists then c will swallow
MANAGE(): swallowing...
IN SWALLOW()
SWALLOW(): org.pwmt.zathura should be swallowing 12:0:bash - "geras"
===========================================================
systemd(1)-+-ModemManager(1278)-+-{ModemManager}(1278)
|-login(33254)---startx(33254)---xinit(33254)-+-Xorg(33293)-+-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | |-{Xorg}(33293)
| | `-{Xorg}(33293)
| `-dwm(33305)-+-compton(33305)-+-{compton}(33305)
| | |-{compton}(33305)
| | |-{compton}(33305)
| | `-{compton}(33305)
| |-dunst(33305)-+-{dunst}(33305)
| | `-{dunst}(33305)
| |-lxpolkit(33305)-+-{lxpolkit}(33305)
| | `-{lxpolkit}(33305)
| |-slstatus(33305)
| |-st(33340)---bash(33341)---screen(33429)---screen(33430)---b+
| |-st(33369)---bash(33370)---pstree(33443)
| |-unclutter(33305)
| `-xautolock(33305)
|-tmux: server(22354)-+-bash(22355)
| |-bash(22513)
| `-bash(24710)
Aha! So we can see that the reason isdescprocess() fails every time a GUI
application is launched from a terminal running tmux is because the GUI
application is launched as a child of the tmux server process which itself is a
child of the init process. There’s no path back up from this new GUI application
to the terminal application, so there’s no way for dwm to determine which window
launched the application and therefore which window should be swallowed.
I also tested this using screen and you can see that screen runs as a child of the terminal emulator window which is why swallowing using screen does still work whereas it doesn’t for tmux.
Summary
dwm’s swallow patch is incapable of handling applications launched from terminal emulators running tmux because the patch figures out which window should be swallowed by finding the parent process of the recently launched GUI application. This fails when running tmux because tmux forks applications from its server process which is a direct child of PID 1 (the init process). There’s no direct path up the process tree from the GUI application to the terminal emulator which means dwm can’t figure out which terminal should be swallowed by the new application so it spawns the application normally.
Swallowing still works with screen because screen is a child process of the terminal emulator and so are applications launched from it. In this case, there is a direct path up from the GUI application to the terminal emulator so dwm can find out which terminal to swallow.
Workaround
Right now, there is an okay-ish workaround to still get window swallowing while
running tmux in the form of the
devour application. It mimics
swallowing using the xdo program but it doesn’t provide a true swallowing
experience because it doesn’t maintain the properties of the window it is
swallowing.
For example, running a GUI application from a floating terminal emulator will spawn the application as a regular tiled window whereas, with dwm’s swallow patch, the GUI application will take over the exact position, size, and properties of the terminal emulator window (as you saw in the first animation at the top of this post).
Conclusion
Here is the build of dwm that I used for testing with source code+printf statements, logs, and pstree outputs. I know it’s a bit messy but I thought I’d provide it anyways in case someone wants it.
Since this behaviour is fundamental to the design of tmux, a modification of the swallow patch would need to be done or an extra dwm patch would need to be created to solve this problem. Looking at the process trees though, I’m not sure how it would be possible to make tmux work with window swallowing.
dwm would somehow have to find out which tmux client corresponds to which application forked from the tmux server process. It seems to me that the approach the swallow patch uses is simply incompatible with the way that tmux works. If there is some kind of tmux API, then there might be a way to interact with the tmux server and client processes to determine which terminal window launched a certain GUI application but I don’t know how tmux’s internals work so this is just a guess. At the moment I don’t have the time to look into fixing this, so if anyone reading this wants to have a go, please feel free (and let me know about it)!
In the meantime, the devour application can be used as a rough replacement for
swallowing if you are dead set on running tmux and want swallowing. For me
though, I’m not too attached to tmux and I won’t be using it. The only tangible
advantage it offered me over my existing workflow was better redrawing of
terminal contents when the terminal gets resized which I can easily live
without.
This is my fifty-fourth post for the #100DaysToOffload challenge. You can learn more about this challenge over at https://100daystooffload.com.