Avahi and ipv6: Need to ping before ssh

SmallAndSimple · 2022-08-07 12:59:22

Recently, my home setup switched to ipv6 because of a router update. I run avahi, and use it to ssh to my laptop. Now, if I just do ssh to my laptop, it first tries ipv6, which times out and then it switches to ipv4:

ssh -vvv laptop.local
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 56: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug2: resolving "laptop.local" port 22
debug3: resolve_host: lookup laptop.local:22
debug3: ssh_connect_direct: entering
debug1: Connecting to laptop.local [2a02:<**>%2] port 22.
debug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48
debug1: connect to address 2a02:<**>%2 port 22: Connection timed out
debug1: Connecting to laptop.local [192.168.2.176] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
etc.

This is slow. If I ping first:

ping laptop.local
PING laptop.local(22a02:<**>%2) 56 data bytes
64 bytes from 22a02:<**>%2: icmp_seq=1 ttl=63 time=3.24 ms

And then ssh:

ssh -vvv laptop.local
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 56: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug2: resolving "laptop.local" port 22
debug3: resolve_host: lookup laptop.local:22
debug3: ssh_connect_direct: entering
debug1: Connecting to laptop.local [2a02:<**>%2] port 22.
debug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48
debug1: Connection established.

It suddenly works! But only in that terminal, if I switch to another terminal, I will have to wait for the timeout again.

What causes this behaviour and what can I do to fix it?

seth · 2022-08-08 22:08:22

if I switch to another terminal, I will have to wait for the timeout again

Smells like resolved, what does your nsswitch.conf look like?
Are you sure ssh and ping (or the subsequent ssh) resolve the exact same IPv6?
Do you use systemd-resolved?
Do you have MulticastDNS=no in resolved.conf(5) ?

bulletmark · 2022-10-20 00:10:25

OP, did you get anywhere with this?

I get the same problem, or at least very similar. I am trying to ssh to the ipv4 address of my Arch based laptop from my phone. I only have ipv4 configured on the wired interface, statically via GNOME Network Manager (ipv6 on that interface, and wifi are disabled). The connection never succeeds and I get sshd message "Connection reset by 192.168.1.89 port 60744 [preauth]". However, if I ping from my phone then I can connect via ssh fine. Later on (not sure of how long or why) then it all happens again. Doubt it is a DNS issue because I am using only the ipv4 address. Same thing happens from a Raspberry Pi. I can't ssh to my laptop until I ping it, and then I can ssh. Phone and Raspberry Pi require their own pings though!

I searched around using Google which led me to this thread, back on the Arch forums where I frequent.

seth · 2022-10-20 06:00:10

Doubt it is a DNS issue

Do you connect by domain or by IP?

Phone and Raspberry Pi require their own pings though!

The OP wrote:

if I switch to another terminal, I will have to wait for the timeout again

Same?

Do you have verbose ssh and sshd logs?

bulletmark · 2022-10-20 13:00:36

seth wrote:

Do you connect by domain or by IP?

I am using the numeric ipv4 address of my Arch laptop sshd server, from both phone and RPi ssh clients.

I said timeout but, as I said, I am not at all sure of what "breaks" it again. Certainly if my laptop suspends and then resumes then both clients need to ping "reset" it again.

sshd merely says

error: kex_exchange_identification: read: Connection reset by peer
Connection reset by 192.168.1.90 port 39482

ssh on the Rpi (Arch Alarm):

pi4:~ ssh -vvv mark@192.168.1.11
OpenSSH_9.1p1, OpenSSL 1.1.1q  5 Jul 2022
debug1: Reading configuration data /home/pi/.ssh/config
debug3: kex names ok: [diffie-hellman-group1-sha1]
debug1: /home/pi/.ssh/config line 198: Applying options for *
debug3: kex names ok: [diffie-hellman-group1-sha1]
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.168.1.11 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/pi/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/pi/.ssh/known_hosts2'
debug1: Control socket "/tmp/ssh-pi@pi4-mark@192.168.1.11:22" does not exist
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.1.11 [192.168.1.11] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/pi/.ssh/id_rsa type 0
debug1: identity file /home/pi/.ssh/id_rsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa type -1
debug1: identity file /home/pi/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519 type -1
debug1: identity file /home/pi/.ssh/id_ed25519-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_xmss type -1
debug1: identity file /home/pi/.ssh/id_xmss-cert type -1
debug1: identity file /home/pi/.ssh/id_dsa type -1
debug1: identity file /home/pi/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.1
kex_exchange_identification: read: Connection reset by peer
Connection reset by 192.168.1.11 port 22

Actually, ssh is irrelevant as it seems to be some kind of lower level network issue. E.g. if I run `python -m http.server` on the laptop then I just get `curl: (56) Recv failure: Connection reset by peer` until I do the ping "fix" then I can get the normal html page.

Actually, I didn't add this previously because it is confusing to explain but when I do the ping "reset" something odd happens. I log in to the headless RPi by ssh from my laptop but I can't ssh back, So I run the ping on that command line and it starts returning normal ping results and then suddenly that ssh connection (i.e. from my laptop to the RPi) dies and I land back on my laptop. Then I ssh back to the RPi and ssh back to the laptop works fine. So the ping "reset" causes my original ssh session to die as it "fixes" the network issue(!).

seth · 2022-10-20 13:15:57

Sounds like a routing/firewall issue that lets ICMP pass and create a context, but drops TCP?
(The NIC apparently doesn't fall asleep, because you need to ping-fix it from various clients individually)

bulletmark · 2022-10-20 22:32:28

@seth, note from the IP addresses I quote in my posts above that these devices are all in the same subnet, so there is no routing between them. There's a couple of switches but that is all.

bulletmark · 2022-10-21 01:38:56

Ok seth's routing suggestion has set me off on some experiments. Below is my annotated record of what I have found. Can anybody explain this?

# Note IP address of lt = 192.168.1.11
# Note IP address of pi2 = 192.168.1.99

# List the routing table on my laptop which is using Network Manager
# (on Arch). Note no direct route for local subnet which seems odds to me?:
lt:~ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG    100    0        0 enp3s0u2u4
192.168.1.254   0.0.0.0         255.255.255.255 UH    100    0        0 enp3s0u2u4

# Do traceroute to RPi and it routes via gateway which even though it is
# on the same subnet appears sensible given above routing table:
lt:~ traceroute 192.168.1.99
traceroute to 192.168.1.99 (192.168.1.99), 30 hops max, 60 byte packets
 1  gateway (192.168.1.254)  0.362 ms  0.346 ms  0.580 ms
 2  pi2 (192.168.1.99)  3.883 ms  3.857 ms  2.302 ms

# Ssh to RPi:
lt:~ ssh pi@192.168.1.99
Last login: Fri Oct 21 10:50:44 2022 from 192.168.1.11

# Note I can not ssh back to laptop due to this bug:
pi2:~ ssh mark@192.168.1.11
kex_exchange_identification: read: Connection reset by peer
Connection reset by 192.168.1.11 port 22

# List the routing table on my RPi which is using systemd-networkd (on Arch
# Alarm). Note has direct route for local subnet as I expect:
pi2:~ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG    10     0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     10     0        0 eth0
192.168.1.254   0.0.0.0         255.255.255.255 UH    10     0        0 eth0

# So I ping my laptop but the ping somehow causes my ssh connection to
# fail and I am dropped back to laptop:
pi2:~ ping 192.168.1.11
PING 192.168.1.11 (192.168.1.11) 56(84) bytes of data.
64 bytes from 192.168.1.11: icmp_seq=1 ttl=63 time=1.02 ms
client_loop: send disconnect: Broken pipe
lt:~

# Back on laptop the routing table has not changed:
lt:~ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG    100    0        0 enp3s0u2u4
192.168.1.254   0.0.0.0         255.255.255.255 UH    100    0        0 enp3s0u2u4

# However a traceroute to RPi is now direct, not via gateway:
lt:~ traceroute 192.168.1.99
traceroute to 192.168.1.99 (192.168.1.99), 30 hops max, 60 byte packets
 1  pi2 (192.168.1.99)  2.578 ms  2.529 ms  2.512 ms

# So I ssh to Rpi and I can now ssh ok back to laptop:
lt:~ ssh pi@192.168.1.99
Last login: Fri Oct 21 11:14:53 2022 from 192.168.1.254
pi2:~ ssh mark@192.168.1.11
Last login: Fri Oct 21 10:50:49 2022 from 192.168.1.99
lt:~ exit

-thc · 2022-10-21 06:02:48

How exactly is the laptops network interface configured?

bulletmark · 2022-10-21 07:15:43

It's just configured statically via the GNOME settings display for Network Manager. Before replying here I decided to "Remove Connection Profile" and then re-add it. I configured it exactly the same wired ipv4 address + netmask + gateway + DNS and with ipv6 disabled. However, now the routing table is:

lt:~ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG    100    0        0 enp3s0u2u4
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 enp3s0u2u4

And now it works fine, as you would expect given those routes, even after rebooting a few times. No idea why that routing is suddenly different given I configured it the same so I will just have to monitor it and see if it breaks again.

seth · 2022-10-21 07:27:16

192.168.1.254   0.0.0.0         255.255.255.255 UH    100    0        0 enp3s0u2u4

would fit some VPN patterns.
Important lesson learned: packages are always routed

bulletmark · 2022-10-21 07:49:58

Sorry seth, can you please explain what you mean? I've never configured a VPN.

seth · 2022-10-21 08:05:27

I asked google what creates these host routes (since it's now gone from the table) and the oracle of moutainview spoke of wireguard and OpenVPN a lot - so I figured there's a good chance that this is what creates them for you.
nb. that your raspi has them as well, next to a regular-ass segment route. Is the systemd-networkd driven network also statically configured there?

bulletmark · 2022-10-21 08:11:52

No all my RPi and all my other devices are just DHCP and then I reserve fixed addresses for them in my router. I have never configured a VPN on any machine in my life. I use ssh for everything.

SmallAndSimple · 2022-10-26 14:59:38

seth wrote:

if I switch to another terminal, I will have to wait for the timeout again
Smells like resolved, what does your nsswitch.conf look like?
Are you sure ssh and ping (or the subsequent ssh) resolve the exact same IPv6?
Do you use systemd-resolved?
Do you have MulticastDNS=no in resolved.conf(5) ?

I am so sorry for only responding now. But here I go. Problem is still there.

q1: is the ipv6 address actually the same? Well, I think so. The ipv6 according to my laptop is 2a02:a45c:<**>:1:b07a:bce3:660c:f82/64 (I think I need to hide the actual ipv6 because it is reachable from the outside world, is that correct?), the ipv6 ssh is trying to connect on is 2a02:a45c:<**>:1:b07a:bce3:660c:f82%2, notice the %2, which is the only difference, but it might not be part of the actual ip? If I try to ping laptop.local, I get 2a02:a45c:<**>:1:b07a:bce3:660c:f82, which is as expected.

q2: Do I use systemd-resolved? No, it is disabled:

sudo systemctl status systemd-resolved                                                                                       130 ↵
○ systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

q3: Do you have MulticastDNS=no in resolved.conf(5)
No, I do not.

I figured out that the problem is specifically ssh over ipv6 to my laptop. I also have a raspberry pi 4, also with ipv6 and there are no issues there (ssh rpi4.local connects instantly via ipv6). My laptop is the only one using networkmanager, but every ipv6 setting is left default.

seth · 2022-10-26 20:09:16

notice the %2, which is the only difference, but it might not be part of the actual ip?

That's the interface ID - what IP does the working ssh (after the ping) address?

Edit: sorry, according to your OP it still addresses the interface.

Edit #2:
Can you ssh the IPv6 (not the domain) w/o having to ping it?
Can you ssh the IPv6 w/o the interface ID ("%2")?

seth wrote:

what does your nsswitch.conf look like?

Last edited by seth (2022-10-26 20:12:32)

SmallAndSimple · 2022-10-27 06:23:42

Ah, I seem to have missed question 0.

nsswitch.conf, on my desktop:

# Name Service Switch configuration file.
# See nsswitch.conf(5) for details.

passwd: files systemd
group: files [SUCCESS=merge] systemd
shadow: files systemd
gshadow: files systemd

publickey: files

hosts: mymachines mdns_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] files myhostname dns
networks: files

protocols: files
services: files
ethers: files
rpc: files

netgroup: files

On my laptop:

# Name Service Switch configuration file.
# See nsswitch.conf(5) for details.

passwd: files systemd
group: files systemd
shadow: files

publickey: files

hosts: files mymachines myhostname mdns_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns
networks: files

protocols: files
services: files
ethers: files
rpc: files

netgroup: files

ssh from my desktop to the laptop ipv6 directly gives a

ssh: connect to host 2a02:a45c:<**>:f82 port 22: Connection timed out

Which simply sounds like there is no one listening on ipv6 port 22?
But ssh from my laptop to the same ipv6 address does work as expected.

Last edited by SmallAndSimple (2022-10-27 06:36:45)

-thc · 2022-10-27 08:21:20

SmallAndSimple wrote:

notice the %2, which is the only difference, but it might not be part of the actual ip?

It's the interface number.

Any IPv6 interface will acquire a "link-local" (fe80::/64) address. Due to the nature of the duplicate address detection (DAD) performed on every network segment (e.g. LAN and WiFi in a notebook attached to both) it is possible for a remote link-local address to exist on more than one segment.

Because of that you have to add (append) the interface number to the address if you want to use the link-local address to reach another host.

A public IPv6 address (2000::/3) on the other hand is unique and adding the interface number should never be necessary.

Your host seems to treat your public IPv6 address like it's link-local - which is rather strange.

seth · 2022-10-27 12:37:00

Can you ping the IPv6 (w/o the identifier) and then ssh the IPv6 (w/o the identifier)
Because of the weird domain handling, does "mdns" instead of "mdns_minimal" cause the same resolution pattern (ping goes to the IPv6 w/o the identifier and ssh to the IPv6 w/ the identifier)?

Also

nsswitch.conf, on my desktop wrote:

hosts: mymachines mdns_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] files myhostname dns

On my laptop wrote:

hosts: files mymachines myhostname mdns_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns

Since we had so much fun w/ the nss-resolve resolution, what if change the hosts cascase on both systems to

files mymachines mdns_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname dns

resp.

files mymachines mdns [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname dns

Arch Linux

#1 2022-08-07 12:59:22

Avahi and ipv6: Need to ping before ssh

#2 2022-08-08 22:08:22

Re: Avahi and ipv6: Need to ping before ssh

#3 2022-10-20 00:10:25

Re: Avahi and ipv6: Need to ping before ssh

#4 2022-10-20 06:00:10

Re: Avahi and ipv6: Need to ping before ssh

#5 2022-10-20 13:00:36

Re: Avahi and ipv6: Need to ping before ssh

#6 2022-10-20 13:15:57

Re: Avahi and ipv6: Need to ping before ssh

#7 2022-10-20 22:32:28

Re: Avahi and ipv6: Need to ping before ssh

#8 2022-10-21 01:38:56

Re: Avahi and ipv6: Need to ping before ssh

#9 2022-10-21 06:02:48

Re: Avahi and ipv6: Need to ping before ssh

#10 2022-10-21 07:15:43

Re: Avahi and ipv6: Need to ping before ssh

#11 2022-10-21 07:27:16

Re: Avahi and ipv6: Need to ping before ssh

#12 2022-10-21 07:49:58

Re: Avahi and ipv6: Need to ping before ssh

#13 2022-10-21 08:05:27

Re: Avahi and ipv6: Need to ping before ssh

#14 2022-10-21 08:11:52

Re: Avahi and ipv6: Need to ping before ssh

#15 2022-10-26 14:59:38

Re: Avahi and ipv6: Need to ping before ssh

#16 2022-10-26 20:09:16

Re: Avahi and ipv6: Need to ping before ssh

#17 2022-10-27 06:23:42

Re: Avahi and ipv6: Need to ping before ssh

#18 2022-10-27 08:21:20

Re: Avahi and ipv6: Need to ping before ssh

#19 2022-10-27 12:37:00

Re: Avahi and ipv6: Need to ping before ssh

Board footer