Cannot ssh from LAN, but can from WAN

Roken · 2020-05-02 08:13:39

This has been driving me nuts for a fortnight.

I have 7 machines on my LAN. Desktop (ARCH), Laptop (ARCH) and 5 x RPi (Raspbian - Debian based).

All have SSH enabled. I can SSH from Desktop to all PIs using local IP (192.168.1.x), and I can SSH from laptop to all machines.

However, none of the PIs nor the desktop can SSH to the laptop using the local IP. However, I can login using the external IP (port forwarded from the router), and I can login to the laptop from the laptop using either IP or localhost.

The error from the LAN is:

sh -vvvv user@192.168.1.xxx
OpenSSH_8.2p1, OpenSSL 1.1.1g  21 Apr 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.168.1.xxx is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.xxx [192.168.1.xxx] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 0
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type 1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2
kex_exchange_identification: read: Connection reset by peer

I've flushed iptables (iptables -F) and there are no rules set up on the target machine.

Chain INPUT (policy ACCEPT 6081 packets, 612K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 5415 packets, 480K bytes)
 pkts bytes target     prot opt in     out     source               destination

journalctl -t sshd (truncated to since last boot)

May 02 08:30:09 TexTop sshd[24363]: Received disconnect from 192.168.1.1 port 59774:11: disconnected by user
May 02 08:30:09 TexTop sshd[24363]: Disconnected from user sshuser 192.168.1.1 port 59774
May 02 08:36:24 TexTop sshd[40909]: Received disconnect from 192.168.1.xxx port 47674:11: disconnected by user
May 02 08:36:24 TexTop sshd[40909]: Disconnected from user sshuser 192.168.1.xxx port 47674
May 02 08:38:44 TexTop sshd[47169]: Received disconnect from ::1 port 41270:11: disconnected by user
May 02 08:38:44 TexTop sshd[47169]: Disconnected from user sshuser ::1 port 41270

EDIT: 192.168.1.1 is my router. The final two entries are the successful logins from localhost and via IP from localhost

I've removed openssh on the laptop (pacman -Rdd openssh), deleted /etc/ssh/ and ~/.ssh/ then reinstalled openssh

I've rebooted probably 100s of times.

openssh is now vanilla.

I'm completely stumped. Any ideas where else to look?

Last edited by Roken (2020-05-02 08:29:52)

ewaller · 2020-05-02 14:42:38

No answers. A few questions.

I assume this is a wireless LAN?
Are you using dynamic addressing? or Static addressing?
If you are using static addressing, did you tell the router somehow?
Can you log in from one RasPi to another?
Have you/ can you tried/try another router?

You might create a wired LAN of your Arch Box, a RasPi; either with one cable, or two cables and an unmanaged switch and try to log in. At least it will give a clue as to which way to look.

Roken · 2020-05-02 16:13:21

No, it's entirely wired.

All IPs are static (they have to be. The RPIs server various functions that I need static addresses for, and I just followed suit with the desktop and laptop)
No problems with ssh from one Pi to another.
This is a new router in the past week. (the problem was there before the router change)

Literally, the only machine with a problem is the laptop (ARCH - still a wired connection) which is simply refusing ssh connections from the LAN, but is happy to accept them from the WAN.

I'm completely flumoxed,

EDIT: And for entirely different reasons, the connection between my desktop and the laptop are, as of 1 hour ago, on a different, unmanaged switch, so that rules the switch out, too.

I should clarify the switch issue since, whilst I don't think it's relevant, I have been accused in the past of missing out information. Until today, my whole network was managed through an 8 port switch. This had four PoE ports, that serviced the four non Pi4B+ machines (the 4B+ is powered from mains). Unfortunately, it's a 100Mbs switch. Today, I've dug out my older 1Gb switch, took the router to that. The laptop, desktop and Pi4B come out of the 1Gb switch, and the PoE switch is fed it's input for the other PIs.

I've verified connection speeds with ethtool across all 7 machines, which reports as expected.

Still no change to behaviour of ssh on the laptop, though. I can ssh from the laptop to other machines on the LAN, but none can get into it.

Last edited by Roken (2020-05-02 16:25:15)

seth · 2020-05-02 20:31:38

Ramp up the server log level, https://wiki.archlinux.org/index.php/Op … leshooting
Whatever the wiki says: do never use "-x" for the journal ;-)

Do you login by password or key?

Roken · 2020-05-03 07:10:58

Ramping up the log level to DEBUG, and there's absolutely nothing in the log (journalctl -e) for ssh.

journalctl -t sshd reports only the successful login from WAN, and not unsuccessful logins from LAN (which is a change since yesterday, but not the right way).

Using password login. I have previously used keys, but not now. However, forcing password with:

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@192.168.1.x

Doesn't help. It isn't getting far enough to ask for the password.

seth · 2020-05-03 08:00:47

Did you restart sshd afterwards (we'll ignore the client side for the moment, because the issue seems to be with the server)

Is anything in your former post supposed to show a failed login in the server log?
("192.168.1.1 is my router. The final two entries are the successful logins from localhost and via IP from localhost" sounded like WAN, localhost, localhost)

Do you forward ssh for all systems? If so, on what ports (on router and local range)?

Roken · 2020-05-03 08:30:18

Actually, looking at that log I believe that only successful connections are being logged. The log file appears to be silent on failed connections.

Yes, all machines are port forwarded from ports ranging in 6,000s to 22 on the local machines, except my desktop which is in the 2,000 range both forwarded and locally.

I didn't just restart sshd, I rebooted the server, to no avail.

tcpdump does confirm that the traffic is arriving at the laptop.

I can post sshd_config if it may help, but as I said, I purged it and so (apart from loglevel) it's default, now.

Just for kicks, I changed the port in sshd_config to see if it helped. No difference in behaviour. (trying to connect to port 22 now results in a "Connection refused" error, as expected, and connecting on the new port is exactly the same error as previously.

Last edited by Roken (2020-05-03 08:36:20)

Lone_Wolf · 2020-05-03 11:18:18

Are the static addresses set locally on each machine or do they get them from a dhcp server ?

Does the issue occur on ipv4 address, ipv6 addresses or both ?

Roken · 2020-05-03 12:59:06

They get the IP addresses from the DHCP server, but I've tried it both ways.

ipv4 only - I don't use ipv6 on the LAN.

EDIT: ip address show confirms that the IP address is correct.

EDIT2: OK, I did a comparison of ip address show between my desktop (no problems) and laptop (problem machine). The only difference between the two is the inclusion of "noprefixroute" on the eth0 on the laptop. I have to admit, I have no idea what that is, or if it will make a difference?

Last edited by Roken (2020-05-03 13:13:41)

seth · 2020-05-03 13:49:58

Actschually™: post the outputs of "ip a" or certainly compare the MTU values - the whole thing starts to smell like a mismatch (which is handled by the NAT but not locally) and unrelated to ssh.
Maybe also try telnet behavior (where, good god, do NOT open that to the interwebz!)

Roken · 2020-05-03 21:42:59

I assume that you mean "ip a" on the server:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:1a:4b:8d:3e:5a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.xxx/24 brd 192.168.1.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 8a:86:31:bd:cc:0f brd ff:ff:ff:ff:ff:ff permaddr 00:1c:bf:71:31:3f

In case I was wrong, MTU 1500 is the same on all machines.

And

telnet -4 192.168.1.xxx 22

logs in no problem.

If I telnet without the port, I get connection refused.

Last edited by Roken (2020-05-03 21:45:28)

seth · 2020-05-04 16:52:52

The important part is that the MTU values are all the same (w/ 1500 being the default value) and since telnet works (telnetd defaults to port 23 so to contact sshd on 22 you'll have to force the port, that's expected) the issue is less likely on that layer but indeed with ssh.
(Though since I've to assume that you did not perform the ssh handshake via telnet, maybe try running telnetd on the problematic system to really login and cause some meaningfull traffic)

There's of course https://bugs.archlinux.org/task/65517 but that should™ be sorted out by a reboot.

Monitor the journal on the ssh server (all of it, no filter) while attempting to ssh into it. Maybe something™ in the ssh handshake makes the NIC crash-restart.
(You might have noticed that we're grasping straws by now…)

Arch Linux

#1 2020-05-02 08:13:39

Cannot ssh from LAN, but can from WAN

#2 2020-05-02 14:42:38

Re: Cannot ssh from LAN, but can from WAN

#3 2020-05-02 16:13:21

Re: Cannot ssh from LAN, but can from WAN

#4 2020-05-02 20:31:38

Re: Cannot ssh from LAN, but can from WAN

#5 2020-05-03 07:10:58

Re: Cannot ssh from LAN, but can from WAN

#6 2020-05-03 08:00:47

Re: Cannot ssh from LAN, but can from WAN

#7 2020-05-03 08:30:18

Re: Cannot ssh from LAN, but can from WAN

#8 2020-05-03 11:18:18

Re: Cannot ssh from LAN, but can from WAN

#9 2020-05-03 12:59:06

Re: Cannot ssh from LAN, but can from WAN

#10 2020-05-03 13:49:58

Re: Cannot ssh from LAN, but can from WAN

#11 2020-05-03 21:42:59

Re: Cannot ssh from LAN, but can from WAN

#12 2020-05-04 16:52:52

Re: Cannot ssh from LAN, but can from WAN

Board footer