tmux killed by systemd when ssh session closed

southerngs · 2014-12-09 03:12:34

I recently installed Arch Linux and I'm having a problem where my tmux sessions gets killed by systemd when I exit the ssh session that was used to start tmux.

Using strace to trace the tmux server process it looks like the server is getting killed by systemd when the ssh session is closed. Below is a snipped showing the signal that is sent.

poll([{fd=4, events=POLLIN}, {fd=11, events=POLLIN}, {fd=6, events=POLLIN}], 3, 497) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---
sendto(3, "\17", 1, 0, NULL, 0)         = 1
+++ killed by SIGKILL +++

I've installed Arch Linux on another system with a very similar configuration and it does not have this problem. I'm new to Arch Linux and systemd so I'm not really sure what I should check to track down why this is happening.

jasonwryan · 2014-12-09 03:29:39

What is the output of `echo $TERM` in and out of tmux on the misbehaving machine?

How do you start tmux on that machine? Is your tmux.conf identical on both machines? Please paste it.

southerngs · 2014-12-09 04:35:24

I start tmux by logging in to the machine using ssh and then typing "tmux". After I disconnect from tmux (typing ctrl-B d) the serve is still running and I can reattach if I want to. But if I close the ssh session then tmux gets killed by systemd.

The user configuration between the two machines is the same because I am logging on with the same account (using ldap) and the home directory is mounted over nfs.

However, on the machine that has the problem I also have the same problem if I log in with root (which is a local account that does not use ldap or nfs). I assume it's probably easiest to troubleshoot this problem with root since it eliminates the nfs/ldap variables.

With root I don't have any tmux.conf file so it will get whatever the defaults are I think.

I ran tmux with verbose logging (-vvv) and I'm including a link to the server log file. I don't think it shows anything interesting, but I've included it just in case:
https://gist.github.com/southerngs/0469b9245c9ea6d8adf6

When I `echo $TERM` the result outside tmux is: "xterm-256color" and inside tmux it is: "screen".

Last edited by southerngs (2014-12-09 05:09:23)

jasonwryan · 2014-12-09 05:06:50

You disconnect from tmux with Prefix-d, not Control-d.

southerngs · 2014-12-09 05:11:54

jasonwryan wrote:

You disconnect from tmux with Prefix-d, not Control-d.

That was a typo. I edited my post to correct it, but yes I use Prefix-d to disconnect from tmux.

jasonwryan · 2014-12-09 06:16:39

Are the same versions of tmux running on both machines? Same operating systems, etc?

southerngs · 2014-12-09 07:05:46

jasonwryan wrote:

Are the same versions of tmux running on both machines? Same operating systems, etc?

Yes both machines have Arch Linux that was just installed during the past week. The version are the same for both machines they are:

Linux 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
tmux 1.9a

There are differences in the hardware of the two machines and possibly some other differences in the configurations (although I followed very similar steps in the install for each of them). The machine that has the problem has UEFI boot, while the one that works correctly is using BIOS boot. But I would not expect that difference to matter, it is just one of the configuration differences between the two machines I can think of.

Also from looking at the traces of what tmux is doing and comparing it with the system that works and the doesn't I don't think the problem is with tmux because the logs and traces look very similar in both cases. I think the problem is with systemd terminating the tmux process on the system that the problem. But I don't know exactly how systemd works so I could be wrong.

jasonwryan · 2014-12-09 07:15:28

If you aren't starting tmux as a service, I don't think systemd can be legitimately blamed here.

The other place to start looking would be outside tmux: the environment in which it is running, anything in the shell initialization files, SSH configs etc that is different between the two boxes.

southerngs · 2014-12-09 17:50:46

jasonwryan wrote:

If you aren't starting tmux as a service, I don't think systemd can be legitimately blamed here.

I don't know exactly what systemd does. The reason that I thought it might be causing the problem is that in the strace trace that I got of the tmux server process, when the process is killed the reason is listed as

--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---

. When I looked at what process had PID 1 (the si_pid) I saw that it is systemd (and systemd seems to always have PID 1). So I thought systemd is trying to kill the tmux process. But I don't know if tmux normally works by catching and handling the signals, or if it is supposed to register itself in some way to prevent systemd from sending it a kill signal when the ssh session is closed.

The other place to start looking would be outside tmux: the environment in which it is running, anything in the shell initialization files, SSH configs etc that is different between the two boxes.

Yeah I'll keep looking and add to the forum if I find anything that looks like a possible cause.

Arch Linux

#1 2014-12-09 03:12:34

tmux killed by systemd when ssh session closed

#2 2014-12-09 03:29:39

Re: tmux killed by systemd when ssh session closed

#3 2014-12-09 04:35:24

Re: tmux killed by systemd when ssh session closed

#4 2014-12-09 05:06:50

Re: tmux killed by systemd when ssh session closed

#5 2014-12-09 05:11:54

Re: tmux killed by systemd when ssh session closed

#6 2014-12-09 06:16:39

Re: tmux killed by systemd when ssh session closed

#7 2014-12-09 07:05:46

Re: tmux killed by systemd when ssh session closed

#8 2014-12-09 07:15:28

Re: tmux killed by systemd when ssh session closed

#9 2014-12-09 17:50:46

Re: tmux killed by systemd when ssh session closed

Board footer