[SOLVED] SSH "No route to host"

felixculpa · 2024-11-04 03:18:01

I recently installed a bunch of updates and I noticed the message about ssh breaking after updating on the main page after. Though I've read just restarting the server should fix that. I've rebooted and I'm having issues still. I've been having weird issues ever since updating, like being unable to open internally hosted webpages with Caddy. For now we'll focus on SSH here.

I can ping my machine just fine but I can't ssh to it. I've tried pinging local machines, the gateway, Google from both the client and the host. That all works. I tried nmap for the first time too. Nmap shows the port is open when I run it on the host, when I try scanning it remotely from the client, it says the ssh port is closed.

Route table on client:

Dst             Gateway         Prefsrc         Protocol Scope   Dev              Table
default         10.0.0.1        10.0.0.7        dhcp             br0
10.0.0.0/28                     10.0.0.7        kernel   link    br0
10.0.0.1                        10.0.0.7        dhcp     link    br0
10.0.10.2       10.0.0.1        10.0.0.7        dhcp             br0
10.0.0.7                        10.0.0.7        kernel   host    br0              local
10.0.0.15                       10.0.0.7        kernel   link    br0              local
127.0.0.0/8                     127.0.0.1       kernel   host    lo               local
127.0.0.1                       127.0.0.1       kernel   host    lo               local
127.255.255.255                 127.0.0.1       kernel   link    lo               local

Client machine is 10.0.0.7, host/server is 10.0.10.2

Edit: Still some weird issues going on, but can SSH in now.

Last edited by felixculpa (2024-11-10 22:48:58)

seth · 2024-11-04 07:45:57

Please use [code][/code] tags, not "quote" tags. Edit your post in this regard.

The service incompatibility won't get you "No route to host", an error which should™ also preclude any ping (because we don't know where to send the ping)

ip r get 10.0.10.2
ping -c1 10.0.10.2
ssh -v 10.0.10.2

it says the ssh port is closed

The server is in a different network segment, does your sshd config allow for that?

felixculpa · 2024-11-05 01:06:06

Output for these commands follow, from the client machine.

ip r get 10.0.10.2

10.0.10.2 via 10.0.0.1 dev br0 src 10.0.0.7 uid 1000
cache

ping -c1 10.0.10.2

PING 10.0.10.2 (10.0.10.2) 56(84) bytes of data.
64 bytes from 10.0.10.2: icmp_seq=1 ttl=63 time=1.61 ms

--- 10.0.10.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.612/1.612/1.612/0.000 ms

ssh -v 10.0.10.2

OpenSSH_9.7p1, OpenSSL 3.3.2 3 Sep 2024
debug1: Reading configuration data /home/<name>/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo-security.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo.conf
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 10.0.10.2 [10.0.10.2] port 22.
debug1: connect to address 10.0.10.2 port 22: No route to host
ssh: connect to host 10.0.10.2 port 22: No route to host

Same output if I specify the port I actually have it configured for.

The network is split up into a few subnets and using vlans, traffic goes between them no problem (except for some firewall/ACL stuff between some of them). Unless sshd has a new option for that, that should be fine.

Last edited by felixculpa (2024-11-05 01:07:19)

seth · 2024-11-05 07:50:54

except for some firewall/ACL stuff between some of them

Wireshark (or tcpdump) the server to see whether the ssh packages actually make it there.
I don't think it's a rejection from sshd and you obviously have a route to the host, so the thing that steps inbetween is a firewall, frequently w/ that symptom.

Common trap is that nftables installes a rudimentary (and very strict) table by defaulf. It would allow traffic on :22, but apparently you're not running sshd on that port…

felixculpa · 2024-11-05 17:51:17

I think it had something to do with DNS resolution? I'm using Blocky as a local DNS proxy and I noticed if I pointed certain subnets at it for DNS it would basically make the network nonfunctional for devices on that network, though it only recently started doing that. If it matters I'm also using systemd-networkd to set a static address on the server/host.

After rebooting the host, I no longer get "no route" messages, but instead "connection refused." Perhaps I need to set up new keys because firewalld shows the correct ports are open.

Also experiencing a weird issue now with certificates, I think port forwarding is now pointing to the wrong device perhaps. Strangely I don't see the host on the topology map via my network management software, it vanished.

Shenanigans I tell you.

seth · 2024-11-05 20:49:30

I think it had something to do with DNS resolution?

Not if you're addressing the server via IP (and likewise use IPs for allowed clients)

but instead "connection refused." Perhaps I need to set up new keys because firewalld shows the correct ports are open.

firewalld was most likely the cause for the "no route to host" situation.
An immediate "connection refused" (w/o any attempt to pass credentials, eg. a password) would happen if there's nothing listening on the port.

nmap -p 22 10.0.10.2 # fix the port

Can you also nmap the server from the bridge/router?

felixculpa · 2024-11-06 05:14:59

I currently made things worse. The server is not getting an IP address, which is really weird because it got one when I plugged it into a different subnet. Will update once I figure that out.

felixculpa · 2024-11-06 18:55:57

Ok, I expanded the DHCP range for that subnet and the server now has an IP address. What's bizarre is I could ping 10.0.10.2 but it's not the server. I checked the arp table from the server and the MAC address for that old IP is pointing at the nearby switch, even though that switch shows it has it's normal IP.

What the actual f*? ?

seth · 2024-11-06 20:24:29

Is anything else attached to the switch? Does the switch respond to random pings if it's the only peer?
Maybe some https://en.wikipedia.org/wiki/Network_a … airpinning ?

felixculpa · 2024-11-07 04:09:26

The switch is connected to a few other devices but they aren't all on at the same time (Apple TV, PS4, and the network Controller device, and the router). I am getting network unreachable error again by the way. I work on IT help desk so this is going to really bother me until it gets resolved

Last edited by felixculpa (2024-11-07 04:10:56)

seth · 2024-11-07 08:57:52

Network unreachable isn't the same as not route to host. And I assume ping isnt affected?

You need to simplify the setup to figure where this is coming from - if you can move this into an isolated, friengly context (cut off the WAN) and therefore afford it, shut down all firewalls, "DNS filters" etc to see whether the basic segment bridging works.
If no, illustrate the network layout (basically "ip -4 a; ip -4 r" on all non-dumb nodes)
If yes, gradually re-introduce complexity until things break again so we can take a close look at that element.

felixculpa · 2024-11-08 01:53:45

I'm thinking of simplifying my network setup after forced to go through all the settings lately. I'm considering having the server directly connected to the router/gateway and do a 1:1 NAT instead of port forwarding as well.

I made a mistake, I said "network unreachable" but I meant ""no route to host." It's doing that again lol.

seth · 2024-11-08 08:28:42

It's doing that again lol.

Extremely most likely firewall/nftables.
Please post the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

And iff you can, disable that and test the behavior w/o.

tomw_ · 2024-11-08 09:48:48

I have this same issue with my odroid c4 , ssh not working last 2 or 3 days.(of course after upgrade) I dont have time to find solusion, and I will make downgrade 4 maybe 5 days back, maybe you can do this same .

Last edited by tomw_ (2024-11-08 11:14:31)

seth · 2024-11-08 13:55:09

https://gitlab.archlinux.org/archlinux/ … mmits/main
The last real version update was > 6 weeks ago, if you're getting "no route to host" that's most likely a firewall as well?

tomw_ · 2024-11-08 15:52:45

I make downgrade to 1.11.2024 and still not working.
Ping is OK , ssh: connect to host 192.168.... port 22: Connection refused
Any ideas ?

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

bluetooth.service | bluetooth.target.wants
cronie.service | multi-user.target.wants
dbus-fi.w1.wpa_supplicant1.service | system
dbus-org.bluez.service | system
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.timesync1.service | system
display-manager.service | system
gcr-ssh-agent.socket | sockets.target.wants
getty@tty1.service | getty.target.wants
iptables-openvpn.service | multi-user.target.wants
NetworkManager.service | multi-user.target.wants
NetworkManager-wait-online.service | network-online.target.wants
openvpn-server@server.service | multi-user.target.wants
p11-kit-server.socket | sockets.target.wants
pipewire-media-session.service | pipewire.service.wants
pipewire-session-manager.service | user
pipewire.socket | sockets.target.wants
pulseaudio.socket | sockets.target.wants
remote-fs.target | multi-user.target.wants
sshd.service | multi-user.target.wants
syncthing@syncuser.service | multi-user.target.wants
systemd-timesyncd.service | sysinit.target.wants
teamviewerd.service | multi-user.target.wants
ufw.service | multi-user.target.wants
vncserver@1.service | multi-user.target.wants
wpa_supplicant.service | multi-user.target.wants
xdg-user-dirs-update.service | default.target.wants

Last edited by tomw_ (2024-11-08 16:18:39)

seth · 2024-11-08 15:58:02

Please use [code][/code] tags. Edit your post in this regard.

iptables-openvpn.service                 | multi-user.target.wants
ufw.service                              | multi-user.target.wants

Next to VPN and firewall - and probably unrelated - you've pipewire and pulseaudio concurrent (=> pipewire-pulse package) and a wpa_supplicant.service colliding w/ NM (the latter starts wpa_supplicant itself)

tomw_ · 2024-11-08 16:24:09

I thing problem its not here. I run ubuntu from pendrive and I have this same mistake

seth · 2024-11-08 16:27:30

Ubuntu for what? The server or the client? Do you even run a properly configured sshd on ubuntu in case it's the server? In case it's the client: the client is likely irrelevant.
For which system is the output in #16?

tomw_ · 2024-11-08 17:06:37

Ok in my case I have armbian - and they crashed something, when I install older version armbian everything now is ok .Now when I ssh from arch to armbian everything looks ok

Last edited by tomw_ (2024-11-08 17:08:47)

felixculpa · 2024-11-09 23:57:05

I was thinking maybe Tomw was onto something but I was getting the same messages from my laptop I believe. Will have to double check. However I'm currently having an issue where I can't get it get an IP address. I even completely reconfigured my network from scratch, everything works except the mini PC. I've tried changing ports, vlans, ethernet cables. I would like to say it's a systemd-networkd misconfiguration, but it gets an address if I plug in the cable that's plugged into my client/workstation. At this point I'm starting to think it might be a hardware issue with the mini PC. I may try Network Manager or something to rule out systemd-networkd misconfiguration.

At least it shows up on my topology map now, but for some reason refuses to grab/accept an IP address from all 16 ports it seems but one.

Last edited by felixculpa (2024-11-09 23:58:41)

felixculpa · 2024-11-10 00:20:43

I'm running it headless and the physical location makes it inconvenient to continously directly connect it to a monitor, but earlier when I was on it I disabled firewalld. I had a hunch so I ran

nft list ruleset

and I noticed it listed a few items, with one of them being it allowing ssh access on port 22. Trying to ssh in on the default port says "connection refused" which feels like a better outcome than "no route to host." There seems to be something funky going on with the firewall. Will be trying to flush the ruleset so it's blank and see what happens after.

seth · 2024-11-10 09:16:42

https://wiki.archlinux.org/title/Nftabl … e_firewall but the service wasn't enabled in your earlier post.

felixculpa · 2024-11-10 18:31:05

Yes it's strange. I never enabled it outright (from what I remember). I know firewalld uses it on the backend though. I think it's definitely a firewall issue.

seth · 2024-11-10 19:35:50

Do you still get tbe bogus/undesired tables?
You could post them, maybe we can figure where this is coming from.

Arch Linux

#1 2024-11-04 03:18:01

[SOLVED] SSH "No route to host"

#2 2024-11-04 07:45:57

Re: [SOLVED] SSH "No route to host"

#3 2024-11-05 01:06:06

Re: [SOLVED] SSH "No route to host"

#4 2024-11-05 07:50:54

Re: [SOLVED] SSH "No route to host"

#5 2024-11-05 17:51:17

Re: [SOLVED] SSH "No route to host"

#6 2024-11-05 20:49:30

Re: [SOLVED] SSH "No route to host"

#7 2024-11-06 05:14:59

Re: [SOLVED] SSH "No route to host"

#8 2024-11-06 18:55:57

Re: [SOLVED] SSH "No route to host"

#9 2024-11-06 20:24:29

Re: [SOLVED] SSH "No route to host"

#10 2024-11-07 04:09:26

Re: [SOLVED] SSH "No route to host"

#11 2024-11-07 08:57:52

Re: [SOLVED] SSH "No route to host"

#12 2024-11-08 01:53:45

Re: [SOLVED] SSH "No route to host"

#13 2024-11-08 08:28:42

Re: [SOLVED] SSH "No route to host"

#14 2024-11-08 09:48:48

Re: [SOLVED] SSH "No route to host"

#15 2024-11-08 13:55:09

Re: [SOLVED] SSH "No route to host"

#16 2024-11-08 15:52:45

Re: [SOLVED] SSH "No route to host"

#17 2024-11-08 15:58:02

Re: [SOLVED] SSH "No route to host"

#18 2024-11-08 16:24:09

Re: [SOLVED] SSH "No route to host"

#19 2024-11-08 16:27:30

Re: [SOLVED] SSH "No route to host"

#20 2024-11-08 17:06:37

Re: [SOLVED] SSH "No route to host"

#21 2024-11-09 23:57:05

Re: [SOLVED] SSH "No route to host"

#22 2024-11-10 00:20:43

Re: [SOLVED] SSH "No route to host"

#23 2024-11-10 09:16:42

Re: [SOLVED] SSH "No route to host"

#24 2024-11-10 18:31:05

Re: [SOLVED] SSH "No route to host"

#25 2024-11-10 19:35:50

Re: [SOLVED] SSH "No route to host"

Board footer