[SOLVED] ssh somestimes does not exit after remote command finished

schard · 2021-12-10 17:00:41

I have a pythons script [1], that I use to parallelize upgrades on remote systems.
Therefor the program uses Python's multiprocessing.Pool object to do n jobs in parallel, as specified.
Sometimes it happens, that the remote pacman command, that is called via ssh, terminates, but the ssh client does not:

1 ✗ neumann@ThinkCentre ~ $ ps auxww | grep ssh
root         451  0.0  0.0   8976  5600 ?        Ss   09:26   0:00 sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups
neumann     1675  0.0  0.0   5952  3856 ?        S    09:27   0:00 /usr/bin/ssh-agent -D -a /run/user/1000/keyring/.ssh
neumann    12222  0.0  0.0  13376  7432 pts/0    S+   16:21   0:00 /usr/bin/ssh -o LogLevel=error -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectTimeout=5 94.terminals.homeinfo.intra /usr/bin/sudo /usr/bin/pacman -Syu --needed --disable-download-timeout --noconfirm
neumann    13528  0.0  0.0   6860  2572 pts/1    R+   17:52   0:00 grep ssh
0 ✓ neumann@ThinkCentre ~ $ termssh 94.terminals.homeinfo.intra pidof pacman
1 ✗ neumann@ThinkCentre ~ $ kill 12222

On the script side, I then see something like:

[ERROR] sysrpc.94: Could not establish SSH connection.

after I killed the stale ssh client.
My question is, why the ssh client sometimes hangs itself up, even though pacman terminated normally, and how I can prevent this from happening.

[1] https://github.com/homeinfogmbh/homeinfotools

Last edited by schard (2021-12-10 20:30:11)

ugjka · 2021-12-10 17:30:58

all .pacnews merged?

schard · 2021-12-10 20:18:40

ugjka wrote:

all .pacnews merged?

On the client, yes. But I fail to see how this would affect ssh.
Here is the excerpt of the respective update from the pacman log of system #94:

[2021-12-10T16:21:14+0100] [PACMAN] Running '/usr/bin/pacman -Syu --needed --disable-download-timeout --noconfirm'
[2021-12-10T16:21:14+0100] [PACMAN] synchronizing package lists
[2021-12-10T16:22:23+0100] [PACMAN] starting full system upgrade
[2021-12-10T16:35:14+0100] [ALPM] running '60-mkinitcpio-remove.hook'...
[2021-12-10T16:35:14+0100] [ALPM] transaction started
[2021-12-10T16:35:14+0100] [ALPM] upgraded filesystem (2021.11.11-1 -> 2021.12.07-1)
[2021-12-10T16:35:14+0100] [ALPM] upgraded bash (5.1.008-1 -> 5.1.012-1)
[2021-12-10T16:35:14+0100] [ALPM] upgraded systemd-libs (249.7-1 -> 249.7-2)
[2021-12-10T16:35:14+0100] [ALPM] upgraded ca-certificates-mozilla (3.72-2 -> 3.73-1)
[2021-12-10T16:35:14+0100] [ALPM] upgraded device-mapper (2.03.14-1 -> 2.03.14-2)
[2021-12-10T16:35:15+0100] [ALPM] upgraded systemd (249.7-1 -> 249.7-2)
[2021-12-10T16:35:23+0100] [ALPM] upgraded dhcpcd (9.4.0-1 -> 9.4.1-1)
[2021-12-10T16:35:23+0100] [ALPM] upgraded harfbuzz (3.1.1-2 -> 3.1.2-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded freetype2 (2.11.0-4 -> 2.11.1-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded lib32-harfbuzz (3.1.1-2 -> 3.1.2-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded lib32-freetype2 (2.11.0-3 -> 2.11.1-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded libcups (1:2.3.3op2-4 -> 1:2.4.0-4)
[2021-12-10T16:35:24+0100] [ALPM] upgraded lib32-libcups (2.3.3-1 -> 2.4.0-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded libdrm (2.4.108-1 -> 2.4.109-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded lib32-libdrm (2.4.108-1 -> 2.4.109-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded sqlite (3.36.0-1 -> 3.37.0-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded nss (3.72-2 -> 3.73-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded lib32-nss (3.72-2 -> 3.73-1)
[2021-12-10T16:35:24+0100] [ALPM] upgraded libx11 (1.7.2-1 -> 1.7.3.1-1)
[2021-12-10T16:35:25+0100] [ALPM] upgraded pango (1:1.48.10-1 -> 1:1.50.0-3)
[2021-12-10T16:35:25+0100] [ALPM] upgraded lib32-pango (1:1.48.10-1 -> 1:1.50.0-3)
[2021-12-10T16:35:25+0100] [ALPM] upgraded lib32-vulkan-icd-loader (1.2.194-1 -> 1.2.202-1)
[2021-12-10T16:35:25+0100] [ALPM] upgraded mkinitcpio-busybox (1.33.1-1 -> 1.34.1-1)
[2021-12-10T16:35:25+0100] [ALPM] upgraded mkinitcpio (30-2 -> 31-2)
[2021-12-10T16:35:28+0100] [ALPM] upgraded linux-lts (5.10.81-1 -> 5.10.84-1)
[2021-12-10T16:35:28+0100] [ALPM] upgraded lvm2 (2.03.14-1 -> 2.03.14-2)
[2021-12-10T16:35:28+0100] [ALPM] upgraded openmpi (4.1.1-3 -> 4.1.2-1)
[2021-12-10T16:35:30+0100] [ALPM] upgraded python (3.9.7-2 -> 3.9.9-1)
[2021-12-10T16:35:30+0100] [ALPM] upgraded python-chardet (4.0.0-2 -> 4.0.0-3)
[2021-12-10T16:35:30+0100] [ALPM] upgraded python-urllib3 (1.26.7-1 -> 1.26.7-2)
[2021-12-10T16:35:30+0100] [ALPM] upgraded python-requests (2.26.0-1 -> 2.26.0-2)
[2021-12-10T16:35:30+0100] [ALPM] upgraded ruby (3.0.2-2 -> 3.0.3-1)
[2021-12-10T16:35:30+0100] [ALPM] upgraded shared-mime-info (2.0+57+gc1d1c70-1 -> 2.0+115+gd74a913-1)
[2021-12-10T16:35:30+0100] [ALPM] upgraded systemd-sysvcompat (249.7-1 -> 249.7-2)
[2021-12-10T16:35:30+0100] [ALPM] upgraded vulkan-icd-loader (1.2.194-1 -> 1.2.202-1)
[2021-12-10T16:35:31+0100] [ALPM] upgraded xfsprogs (5.14.0-1 -> 5.14.2-1)
[2021-12-10T16:35:31+0100] [ALPM] upgraded xorg-xauth (1.1-2 -> 1.1.1-1)
[2021-12-10T16:35:31+0100] [ALPM] upgraded xxhash (0.8.0-1 -> 0.8.1-1)
[2021-12-10T16:35:53+0100] [ALPM] transaction completed
[2021-12-10T16:35:54+0100] [ALPM] running '20-systemd-sysusers.hook'...
[2021-12-10T16:35:54+0100] [ALPM] running '30-systemd-catalog.hook'...
[2021-12-10T16:35:54+0100] [ALPM] running '30-systemd-daemon-reload.hook'...
[2021-12-10T16:35:54+0100] [ALPM] running '30-systemd-hwdb.hook'...
[2021-12-10T16:35:56+0100] [ALPM] running '30-systemd-sysctl.hook'...
[2021-12-10T16:35:56+0100] [ALPM] running '30-systemd-tmpfiles.hook'...
[2021-12-10T16:35:56+0100] [ALPM] running '30-systemd-udev-reload.hook'...
[2021-12-10T16:35:56+0100] [ALPM] running '30-systemd-update.hook'...
[2021-12-10T16:35:56+0100] [ALPM] running '30-update-mime-database.hook'...
[2021-12-10T16:35:57+0100] [ALPM] running '40-update-ca-trust.hook'...
[2021-12-10T16:35:59+0100] [ALPM] running '60-depmod.hook'...
[2021-12-10T16:36:13+0100] [ALPM] running '90-mkinitcpio-install.hook'...
[2021-12-10T16:36:14+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
[2021-12-10T16:36:14+0100] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
[2021-12-10T16:36:14+0100] [ALPM-SCRIPTLET] ==> Starting build: 5.10.84-1-lts
[2021-12-10T16:36:14+0100] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2021-12-10T16:36:15+0100] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2021-12-10T16:36:15+0100] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2021-12-10T16:36:15+0100] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2021-12-10T16:36:15+0100] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2021-12-10T16:36:16+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2021-12-10T16:36:17+0100] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2021-12-10T16:36:17+0100] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2021-12-10T16:36:18+0100] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2021-12-10T16:36:19+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2021-12-10T16:36:19+0100] [ALPM-SCRIPTLET] ==> Creating uncompressed initcpio image: /boot/initramfs-linux-lts.img
[2021-12-10T16:36:19+0100] [ALPM-SCRIPTLET] ==> Image generation successful
[2021-12-10T16:36:19+0100] [ALPM] running '99-update-systemd-boot.hook'...
[2021-12-10T16:36:19+0100] [ALPM] running 'dbus-reload.hook'...
[2021-12-10T16:36:19+0100] [ALPM] running 'texinfo-install.hook'...

No issues there.

Solution
After some more googling, I found the answer on SO and here.

Last edited by schard (2021-12-10 20:34:39)

Arch Linux

#1 2021-12-10 17:00:41

[SOLVED] ssh somestimes does not exit after remote command finished

#2 2021-12-10 17:30:58

Re: [SOLVED] ssh somestimes does not exit after remote command finished

#3 2021-12-10 20:18:40

Re: [SOLVED] ssh somestimes does not exit after remote command finished

Board footer