SSH suddenly timing out?

gunnarniels · 2022-02-08 15:38:50

I'm having a strange issue that after years of functionality my sshd has suddenly
stopped working on my workstation and connections from my laptop time out.
I'm able to ping the machine. I've increased the log level on the client and
server and don't see anything notable. I don't run a firewall on my workstation
(although I suppose it's possible I've got something running I don't expect).
I've traced the packets with an iptable rule, so I can see them entering the
destination machine's networking stack. From my reading of packet flows, it looks
like they are actually reaching the daemon, but maybe someone can point out something
that I'm missing. nmap appears to view port 22 as filtered

Any ideas for how to figure out what's blocking this?

# sudo nmap -sS -p22 <dst-ip>
Starting Nmap 7.92 ( https://nmap.org ) at 2022-02-08 10:36 EST
Nmap scan report for <dst-hostname> (<dst-ip>)
Host is up (0.0026s latency).

PORT   STATE    SERVICE
22/tcp filtered ssh
MAC Address: <snip>

Nmap done: 1 IP address (1 host up) scanned in 0.47 seconds

Verbose client logs:

# ssh -v <hostname>
OpenSSH_8.8p1, OpenSSL 1.1.1m  14 Dec 2021
debug1: Reading configuration data /home/<userhome>/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to <hostname> [<dst-ip>] port 22.
debug1: connect to address <dst-ip> port 22: Connection timed out
ssh: connect to host <hostname> port 22: Connection timed out

IP tables trace
Feb 08 09:34:49 <snip> kernel: TRACE: raw:PREROUTING:policy:2 IN=enp0s25 OUT= MAC=<snip> SRC=<src-ip> DST=<dest-ip> LEN=60 TOS=0x08 PREC=0x40 TTL=64 ID=34611 DF PROTO=TCP SPT=37942 DPT=22 SEQ=244534573 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A608D9D470000000001030307)

Feb 08 09:34:49 <snip> kernel: TRACE: mangle:PREROUTING:policy:1 IN=enp0s25 OUT= MAC=<snip> SRC=<src-ip> DST=<dest-ip> LEN=60 TOS=0x08 PREC=0x40 TTL=64 ID=34611 DF PROTO=TCP SPT=37942 DPT=22 SEQ=244534573 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A608D9D470000000001030307)

Feb 08 09:34:49 <snip> kernel: TRACE: mangle:INPUT:policy:1 IN=enp0s25 OUT= MAC=<snip> SRC=<src-ip> DST=<dest-ip> LEN=60 TOS=0x08 PREC=0x40 TTL=64 ID=34611 DF PROTO=TCP SPT=37942 DPT=22 SEQ=244534573 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A608D9D470000000001030307)

Feb 08 09:34:49 <snip> kernel: TRACE: filter:INPUT:policy:1 IN=enp0s25 OUT= MAC=<snip> SRC=<src-ip> DST=<dest-ip> LEN=60 TOS=0x08 PREC=0x40 TTL=64 ID=34611 DF PROTO=TCP SPT=37942 DPT=22 SEQ=244534573 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A608D9D470000000001030307)

Feb 08 09:34:49 <snip> kernel: TRACE: security:INPUT:policy:1 IN=enp0s25 OUT= MAC=<snip> SRC=<src-ip> DST=<dest-ip> LEN=60 TOS=0x08 PREC=0x40 TTL=64 ID=34611 DF PROTO=TCP SPT=37942 DPT=22 SEQ=244534573 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A608D9D470000000001030307)

LogLevel=DEBUG
$ journalctl -b -u sshd

Feb 08 09:19:17 <hostname> systemd[1]: Started OpenSSH Daemon.
Feb 08 09:19:17 <hostname> sshd[12796]: debug1: Set /proc/self/oom_score_adj from 0 to -1000
Feb 08 09:19:17 <hostname> sshd[12796]: debug1: Bind to port 22 on 0.0.0.0.
Feb 08 09:19:17 <hostname> sshd[12796]: Server listening on 0.0.0.0 port 22.
Feb 08 09:19:17 <hostname> sshd[12796]: debug1: Bind to port 22 on ::.
Feb 08 09:19:17 <hostname> sshd[12796]: Server listening on :: port 22.

[root@<dst-hostname> ~]# iptables -nL -t security
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

tucuxi · 2022-02-08 18:10:21

What can we learn from the following?

ssh -Tvvv secret-hostname

gunnarniels · 2022-02-08 19:53:59

Some more odd details: I have a wireless card in the machine. I used nmcli to bring down the wired line and to bring up the wireless connection.

Sure enough nmap shows ssh as open and I'm able to ssh into this interface. I recently had a hard cat6 line installed in my house for this machine.
It has me wondering if something is actually wrong at the physical layer. I'm going to attempt to connect it through a different cable and switch and
see if I see more odd behavior. Another anecdote, I've seen my machine locking up occasionally and some difficulties with my NFS mount. If I SSH from
this machine over the hardwire, sometimes it will lock up as well.

# ssh -Tvvv <dst-hostname>
OpenSSH_8.8p1, OpenSSL 1.1.1m  14 Dec 2021
debug1: Reading configuration data /home/<user>/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname <dst-hostname> is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/<user>/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/<user>/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to <dst-hostname> [<dst-hostname>] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: connect to address <dst-hostname> port 22: Connection timed out
ssh: connect to host <dst-hostname> port 22: Connection timed out

Last edited by gunnarniels (2022-02-08 19:54:39)

daring · 2022-02-09 06:24:40

I have the same problem since yesterday. Can't find the root cause, but have found the workaround: setting

Host *
IPQoS 0x00

in she ~/.ssh/config file did the trick for me.

gunnarniels · 2022-02-09 13:31:00

I'll give that a try and see if anything improves, thanks. Useful to hear at least another data point that someone has seen similar behavior.

tucuxi · 2022-02-09 18:07:45

I suspect your issue is more fundamental as you cannot even establish a TCP connection. Are you sure you are not using any firewall, given that you seem to play around with iptables?

seth · 2022-02-09 22:58:59

Different network segment, MTU issue ("ip a" for wired and wireless connection) and also see https://wiki.archlinux.org/title/Nftables#List_tables because firewalld uses that and they won't show up in iptables.
Can you ping the ssh server on either connection? "ping -s 1480"?

Arch Linux

#1 2022-02-08 15:38:50

SSH suddenly timing out?

#2 2022-02-08 18:10:21

Re: SSH suddenly timing out?

#3 2022-02-08 19:53:59

Re: SSH suddenly timing out?

#4 2022-02-09 06:24:40

Re: SSH suddenly timing out?

#5 2022-02-09 13:31:00

Re: SSH suddenly timing out?

#6 2022-02-09 18:07:45

Re: SSH suddenly timing out?

#7 2022-02-09 22:58:59

Re: SSH suddenly timing out?

Board footer