Table of Contents

Why dwm’s Window Swallowing Patch Can’t Swallow tmux

Note: Code snippets in this blog post are governed by the MIT/X Consortium License as they are snippets taken from dwm code. See the dwm LICENSE file.

The dwm swallow patch is my favourite patch for dwm because it allows me to launch GUI applications from my terminal and have the launched application take over the terminal instead of spawning next to the terminal and rendering the terminal window useless. You can sort of do this by just appending & and then closing the terminal, but you also don’t get the terminal back once the application has closed. Something like launching a video with mpv <video> and having the mpv window replace the terminal window is very nice.

I was also playing around with running every terminal window with tmux specifically because it would allow me to remove the scrollback patch from my terminal emulator st and becuse it handles redrawing the text when the terminal window is resized. However, I noticed that launching applications from within a terminal running tmux wouldn’t get swallowed. A brief search on the interwebs turned up no useful information, so I decided to get my hands dirty and investigate why this was the case.

Before we get started: yes, I know I should learn to use gdb. I have used it before but frequently find printfs do a good enough job that I don’t have to faff about with gdb.

Jump down to the summary to skip the journey and go right to the explanation.

Demonstration

Take a look at the animations below to see what happens when a GUI application is launched while tmux is running:

Click for a higher resolution version of the above video.

Click for a higher resolution version of the above video.

So, what’s the difference between a normal terminal launching a GUI application and a terminal running tmux launching a GUI application?

By this point, I had no idea if the actual cause was something wrong with my configuration of dwm. Perhaps my recent conversion from the scratchpad to the named-scratchpads patch messed something up so I started by looking at the values of the rules[] array in config.h to see if my terminal window was being wrongly interpreted as being unable to swallow:

static const Rule rules[] = {
    /* class     instance  title           tags mask  isfloating  isterminal  noswallow  monitor  scratch key*/
    { "Galculator", NULL,  NULL,           0,         1,          0,          -1,        -1,       0  },
    { "Gimp",    NULL,     NULL,           0,         0,          0,          -1,        -1,       0  },
    { "Firefox", NULL,     NULL,           1 << 1,    0,          0,          -1,        -1,       0  },
    { "St",      NULL,     NULL,           0,         0,          1,           0,        -1,       0  },
    { NULL,      NULL,     "scratchpad",   0,         1,          1,           1,        -1,      's' },
    { NULL,      NULL,     "calculator",   0,         1,          1,           1,        -1,      'c' },
    { NULL,      NULL,     "Event Tester", 0,         1,          0,           1,        -1,       0  },
};

These rules are interpreted by the function applyrules() which applies rules to a launched client based on what’s in the rules[] array. A few printfs later and I had found out that everything was being correctly interpreted here.

Finding The Answer

Since that thread didn’t lead anywhere, I examined the content of the swallow patch itself to see what code it added and where I should look next. It led me to look inside the manage() function which calls swallow() at the very end if the variable term is a truthy value.

void
manage(Window w, XWindowAttributes *wa)
{
    ...
    if (term)
        swallow(term, c);
    focus(NULL);
}

term is of type Client * and is set to NULL at the beginning of the function:

void
manage(Window w, XWindowAttributes *wa)
{
    Client *c, *t = NULL, *term = NULL;
    ...
}

term is then later set by the termforwin() function:

void
manage(Window w, XWindowAttributes *wa)
{
    ...
    if (XGetTransientForHint(dpy, w, &trans) && (t = wintoclient(trans))) {
        c->mon = t->mon;
        c->tags = t->tags;
    } else {
        c->mon = selmon;
        applyrules(c);
        term = termforwin(c);
    }
    ...
}

termforwin() returns a Client * value which represents the terminal which launched the client application. It determines which terminal window the GUI application was just launched from so dwm knows which window should be swallowed.

My printfs up to this point told me that termforwin() was returning NULL when a GUI application was launched inside of a terminal running tmux which is why swallow() was not being called for these applications:

Applying rules to client st
Rules for window:
Title:      (null)
Class:      St
Instance:   (null)
IsTerminal: 1
NoSwallow:  0
IsFloating: 0
Tags:       0
Scratchkey: 
IN MANAGE(): term: (nil)
Applying rules to client org.pwmt.zathura
IN MANAGE(): term: 0x5626932ad110                        <--- No tmux
Entered swallow function
IN SWALLOW(): org.pwmt.zathura should be swallowing st
Applying rules to client org.pwmt.zathura
IN MANAGE(): term: (nil)                                 <--- With tmux

Now I had to take a look inside termforwin() to discover why it was returning NULL for these windows. Since there were only two places where NULL could be returned, I checked which of the two conditions were failing. It turned out that the second return NULL; was being reached which meant that the if statement inside the for loops was evaluating to false for every value of c.

Client *
termforwin(const Client *w)
{
    Client *c;
    Monitor *m;

    if (!w->pid || w->isterminal)
        return NULL;

    for (m = mons; m; m = m->next) {
        for (c = m->clients; c; c = c->next) {
            if (c->isterminal && !c->swallowing && c->pid && isdescprocess(c->pid, w->pid))
                return c;
        }
    }

    return NULL;
}

A few more printfs later and I discovered that the cause of the failure was that isdescprocess() was returning 0. isdescprocess() is a function that determines if a process is a descendant of another process by walking up the process tree. In the case above, it is trying to determine if the next client c in the list of clients is a parent process of our newly launched process w.

This check consistently failed every time a GUI application was launched from a terminal running tmux which meant that the variable term was always NULL. To get a closer look at why this might be failing, I added some more printfs in strategic locations and ran pstree -g alongside every test I performed.

Here are the three scenarios I tested with relevant sections from the output log I created (dwm.log.6) along with the relevant snippets from the pstree output:

Zathura with Just st

MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client st
TERMFORWIN(): pid: 33340, isterminal: 1
MANAGE(): term: (nil)
MANAGE(): if term exists then c will swallow
MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client st
TERMFORWIN(): pid: 33369, isterminal: 1
MANAGE(): term: (nil)
MANAGE(): if term exists then c will swallow
MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client org.pwmt.zathura
TERMFORWIN(): pid: 33382, isterminal: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33369, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33340, isdescproc: 33340
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33382
ISDESCPROCESS(): pid 33382
ISDESCPROCESS(): pid 33341
MANAGE(): term: 0x55b88574e0e0
MANAGE(): if term exists then c will swallow
MANAGE(): swallowing...
IN SWALLOW()
SWALLOW(): org.pwmt.zathura should be swallowing st
=======================================================
systemd(1)-+-ModemManager(1278)-+-{ModemManager}(1278)
           |-login(33254)---startx(33254)---xinit(33254)-+-Xorg(33293)-+-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             `-{Xorg}(33293)
           |                                             `-dwm(33305)-+-compton(33305)-+-{compton}(33305)
           |                                                          |                |-{compton}(33305)
           |                                                          |                |-{compton}(33305)
           |                                                          |                `-{compton}(33305)
           |                                                          |-dunst(33305)-+-{dunst}(33305)
           |                                                          |              `-{dunst}(33305)
           |                                                          |-lxpolkit(33305)-+-{lxpolkit}(33305)
           |                                                          |                 `-{lxpolkit}(33305)
           |                                                          |-slstatus(33305)
           |                                                          |-st(33340)---bash(33341)---zathura(33382)-+-{zathura}(33382)
           |                                                          |                                          |-{zathura}(33382)
           |                                                          |                                          |-{zathura}(33382)
           |                                                          |                                          |-{zathura}(33382)
           |                                                          |                                          |-{zathura}(33382)
           |                                                          |                                          |-{zathura}(33382)
           |                                                          |                                          `-{zathura}(33382)
           |                                                          |-st(33369)---bash(33370)---pstree(33390)
           |                                                          |-unclutter(33305)
           |                                                          `-xautolock(33305)

Zathura with st Running tmux

MANAGE(): XGetTransientForHint: 0
MANAGE(): wintoclient: (nil)
APPLYRULES(): applying to client org.pwmt.zathura
TERMFORWIN(): pid: 33414, isterminal: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33369, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33340, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33414
ISDESCPROCESS(): pid 33414
ISDESCPROCESS(): pid 33398
ISDESCPROCESS(): pid 22354
TERMFORWIN(): Passed pid and isterminal check but returning NULL
MANAGE(): term: (nil)
MANAGE(): if term exists then c will swallow
===========================================================
systemd(1)-+-ModemManager(1278)-+-{ModemManager}(1278)
           |-login(33254)---startx(33254)---xinit(33254)-+-Xorg(33293)-+-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             `-{Xorg}(33293)
           |                                             `-dwm(33305)-+-compton(33305)-+-{compton}(33305)
           |                                                          |                |-{compton}(33305)
           |                                                          |                |-{compton}(33305)
           |                                                          |                `-{compton}(33305)
           |                                                          |-dunst(33305)-+-{dunst}(33305)
           |                                                          |              `-{dunst}(33305)
           |                                                          |-lxpolkit(33305)-+-{lxpolkit}(33305)
           |                                                          |                 `-{lxpolkit}(33305)
           |                                                          |-slstatus(33305)
           |                                                          |-st(33340)---bash(33341)---tmux: client(33397)
           |                                                          |-st(33369)---bash(33370)---pstree(33418)
           |                                                          |-unclutter(33305)
           |                                                          `-xautolock(33305)
           |-tmux: server(22354)-+-bash(22355)
           |                     |-bash(22513)
           |                     |-bash(24710)
           |                     `-bash(33398)---zathura(33414)-+-{zathura}(33414)
           |                                                    |-{zathura}(33414)
           |                                                    `-{zathura}(33414)

Zathura with st Running screen

APPLYRULES(): applying to client org.pwmt.zathura
TERMFORWIN(): pid: 33439, isterminal: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33369, isdescproc: 0
ISDESCPROCESS(): Walking up the process tree to find if p 33369 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
ISDESCPROCESS(): pid 33340
ISDESCPROCESS(): pid 33305
ISDESCPROCESS(): pid 33292
ISDESCPROCESS(): pid 33259
ISDESCPROCESS(): pid 33254
ISDESCPROCESS(): pid 1
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
TERMFORWIN(): isterm: 1, swallowing: (nil), pid: 33340, isdescproc: 33340
ISDESCPROCESS(): Walking up the process tree to find if p 33340 is parent of c 33439
ISDESCPROCESS(): pid 33439
ISDESCPROCESS(): pid 33431
ISDESCPROCESS(): pid 33430
ISDESCPROCESS(): pid 33429
ISDESCPROCESS(): pid 33341
MANAGE(): term: 0x55b88574e0e0
MANAGE(): if term exists then c will swallow
MANAGE(): swallowing...
IN SWALLOW()
SWALLOW(): org.pwmt.zathura should be swallowing 12:0:bash - "geras"
===========================================================
systemd(1)-+-ModemManager(1278)-+-{ModemManager}(1278)
           |-login(33254)---startx(33254)---xinit(33254)-+-Xorg(33293)-+-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             |-{Xorg}(33293)
           |                                             |             `-{Xorg}(33293)
           |                                             `-dwm(33305)-+-compton(33305)-+-{compton}(33305)
           |                                                          |                |-{compton}(33305)
           |                                                          |                |-{compton}(33305)
           |                                                          |                `-{compton}(33305)
           |                                                          |-dunst(33305)-+-{dunst}(33305)
           |                                                          |              `-{dunst}(33305)
           |                                                          |-lxpolkit(33305)-+-{lxpolkit}(33305)
           |                                                          |                 `-{lxpolkit}(33305)
           |                                                          |-slstatus(33305)
           |                                                          |-st(33340)---bash(33341)---screen(33429)---screen(33430)---b+
           |                                                          |-st(33369)---bash(33370)---pstree(33443)
           |                                                          |-unclutter(33305)
           |                                                          `-xautolock(33305)
           |-tmux: server(22354)-+-bash(22355)
           |                     |-bash(22513)
           |                     `-bash(24710)

Aha! So we can see that the reason isdescprocess() fails every time a GUI application is launched from a terminal running tmux is because the GUI application is launched as a child of the tmux server process which itself is a child of the init process. There’s no path back up from this new GUI application to the terminal application, so there’s no way for dwm to determine which window launched the application and therefore which window should be swallowed.

I also tested this using screen and you can see that screen runs as a child of the terminal emulator window which is why swallowing using screen does still work whereas it doesn’t for tmux.

Summary

dwm’s swallow patch is incapable of handling applications launched from terminal emulators running tmux because the patch figures out which window should be swallowed by finding the parent process of the recently launched GUI application. This fails when running tmux because tmux forks applications from its server process which is a direct child of PID 1 (the init process). There’s no direct path up the process tree from the GUI application to the terminal emulator which means dwm can’t figure out which terminal should be swallowed by the new application so it spawns the application normally.

Swallowing still works with screen because screen is a child process of the terminal emulator and so are applications launched from it. In this case, there is a direct path up from the GUI application to the terminal emulator so dwm can find out which terminal to swallow.

Workaround

Right now, there is an okay-ish workaround to still get window swallowing while running tmux in the form of the devour application. It mimics swallowing using the xdo program but it doesn’t provide a true swallowing experience because it doesn’t maintain the properties of the window it is swallowing.

For example, running a GUI application from a floating terminal emulator will spawn the application as a regular tiled window whereas, with dwm’s swallow patch, the GUI application will take over the exact position, size, and properties of the terminal emulator window (as you saw in the first animation at the top of this post).

Conclusion

Here is the build of dwm that I used for testing with source code+printf statements, logs, and pstree outputs. I know it’s a bit messy but I thought I’d provide it anyways in case someone wants it.

Since this behaviour is fundamental to the design of tmux, a modification of the swallow patch would need to be done or an extra dwm patch would need to be created to solve this problem. Looking at the process trees though, I’m not sure how it would be possible to make tmux work with window swallowing.

dwm would somehow have to find out which tmux client corresponds to which application forked from the tmux server process. It seems to me that the approach the swallow patch uses is simply incompatible with the way that tmux works. If there is some kind of tmux API, then there might be a way to interact with the tmux server and client processes to determine which terminal window launched a certain GUI application but I don’t know how tmux’s internals work so this is just a guess. At the moment I don’t have the time to look into fixing this, so if anyone reading this wants to have a go, please feel free (and let me know about it)!

In the meantime, the devour application can be used as a rough replacement for swallowing if you are dead set on running tmux and want swallowing. For me though, I’m not too attached to tmux and I won’t be using it. The only tangible advantage it offered me over my existing workflow was better redrawing of terminal contents when the terminal gets resized which I can easily live without.

This is my fifty-fourth post for the #100DaysToOffload challenge. You can learn more about this challenge over at https://100daystooffload.com.