You are not logged in.
Hi, all!
I've been attempting to fix this issue for the past few hours. Every time I reboot this computer sshd doesn't start, which I always notice when trying to access it from my laptop, then I walk of shame right onto my chair and run "sudo systemctl start sshd" to start it up.
I just rebooted the machine to get some useful debugging output:
❯ sudo systemctl status sshd.service
● sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
Active: inactive (dead)
❯ journalctl -u sshd | tail -n 10
Jul 13 17:54:49 void systemd[1]: Stopping OpenSSH Daemon...
Jul 13 17:54:49 void systemd[1]: sshd.service: Succeeded.
Jul 13 17:54:49 void systemd[1]: Stopped OpenSSH Daemon.
-- Reboot --
Jul 14 10:57:17 void systemd[1]: Started OpenSSH Daemon.
Jul 14 10:57:17 void sshd[11416]: Server listening on 0.0.0.0 port 22.
Jul 14 10:57:17 void sshd[11416]: Server listening on :: port 22.
Jul 14 10:59:35 void systemd[1]: Stopping OpenSSH Daemon...
Jul 14 10:59:35 void systemd[1]: sshd.service: Succeeded.
Jul 14 10:59:35 void systemd[1]: Stopped OpenSSH Daemon.So, nothing useful there. Let's see what the unit does:
❯ cat /etc/systemd/system/sshd@.service
[Unit]
Description=OpenSSH Per-Connection Daemon
After=sshdgenkeys.service
[Service]
ExecStart=-/usr/bin/sshd -i
StandardInput=socket
StandardError=syslog
KillMode=processAlright, so there's a dependency. Let's look into it:
❯ sudo systemctl status sshdgenkeys.service
● sshdgenkeys.service - SSH Key Generation
Loaded: loaded (/usr/lib/systemd/system/sshdgenkeys.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Tue 2020-07-14 10:59:59 -03; 1min 19s ago
├─ ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key.pub was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key.pub was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key.pub was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key was not met
└─ ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key.pub was not met
Jul 14 10:59:59 void systemd[1]: Condition check resulted in SSH Key Generation being skipped.
❯ ls -la /etc/ssh
Permissions Size User Group Date Modified Name
.rw-r--r-- 525k root root 27 May 3:57 moduli
.rw-r--r-- 1.5k root root 27 May 3:57 ssh_config
.rw------- 1.4k root root 14 Jul 10:39 ssh_host_dsa_key
.rw-r--r-- 599 root root 14 Jul 10:39 ssh_host_dsa_key.pub
.rw------- 505 root root 14 Jul 10:39 ssh_host_ecdsa_key
.rw-r--r-- 171 root root 14 Jul 10:39 ssh_host_ecdsa_key.pub
.rw------- 399 root root 14 Jul 10:39 ssh_host_ed25519_key
.rw-r--r-- 91 root root 14 Jul 10:39 ssh_host_ed25519_key.pub
.rw------- 2.6k root root 14 Jul 10:39 ssh_host_rsa_key
.rw-r--r-- 563 root root 14 Jul 10:39 ssh_host_rsa_key.pub
.rw-r--r-- 3.1k root root 14 Jul 10:59 sshd_configSo the key files are there, so I don't understand what else could be wrong. Here's a diff from the stock `sshd_config` and mine:
~
❯ diff /etc/ssh/{sshd_config,sshd_config.mine}
13c13
< #Port 22
---
> Port 31415
32c32
< #PermitRootLogin prohibit-password
---
> PermitRootLogin no
57,58c57,58
< #PasswordAuthentication yes
< #PermitEmptyPasswords no
---
> PasswordAuthentication no
> PermitEmptyPasswords noAnd, finally, manually starting `sshd`:
~
❯ sudo systemctl start sshd
~
❯ sudo systemctl status sshd
● sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-07-14 11:13:08 -03; 2s ago
Main PID: 5488 (sshd)
Tasks: 1 (limit: 19052)
Memory: 832.0K
CGroup: /system.slice/sshd.service
└─5488 sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups
Jul 14 11:13:08 void systemd[1]: Started OpenSSH Daemon.
Jul 14 11:13:08 void sshd[5488]: Server listening on 0.0.0.0 port 31415.
Jul 14 11:13:08 void sshd[5488]: Server listening on :: port 31415.
❯ sudo systemctl status sshdgenkeys.service
● sshdgenkeys.service - SSH Key Generation
Loaded: loaded (/usr/lib/systemd/system/sshdgenkeys.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Tue 2020-07-14 11:13:08 -03; 21s ago
├─ ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key.pub was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key.pub was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key.pub was not met
├─ ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key was not met
└─ ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key.pub was not met
Jul 14 10:59:59 void systemd[1]: Condition check resulted in SSH Key Generation being skipped.
Jul 14 11:11:50 void systemd[1]: Condition check resulted in SSH Key Generation being skipped.
Jul 14 11:13:08 void systemd[1]: Condition check resulted in SSH Key Generation being skipped.All in all, I don't understand what could be the problem here. If I can start sshd manually even if sshdgenkeys still fails, what am I missing here? The service is enabled, the files are there, the sshd_config file is barely modified from stock... I'd appreciate any help on this!
Hope everyone's having a good week.
SOLUTION
Check if sshd.socket is enabled with "systemctl status sshd.socket". If it is, disable it and enable "sshd.service" (if you haven't yet). That should be it.
Last edited by efficientbug (2020-07-14 19:05:57)
Offline
Have you tried
sudo syatemctl enable sshd.serviceOffline
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
Active: inactive (dead)Shows the service was enabled.
cat /etc/systemd/system/sshd@.service
[Unit]
Description=OpenSSH Per-Connection Daemon
After=sshdgenkeys.service
[Service]
ExecStart=-/usr/bin/sshd -i
StandardInput=socket
StandardError=syslog
KillMode=processIs for the no longer shipped sshd@.service. Is sshd@ enabled as well as sshd?
What is the default systemd target of the system? Please post the journal for a boot with the issue.
Offline
Have you tried
sudo syatemctl enable sshd.service
It's enabled, the output for "systemctl status sshd.service" is up there: "Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)".
Is for the no longer shipped sshd@.service. Is sshd@ enabled as well as sshd?
What is the default systemd target of the system? Please post the journal for a boot with the issue.
My bad, I got the contents from the wrong service file, sshd@ is not enabled.
❯ sudo systemctl status sshd@.service
Failed to get properties: Unit name sshd@.service is neither a valid invocation ID nor unit name.Regarding the systemd target, do you mean graphical.target? I only know how to check that with `systemd-analyze`:
❯ systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
graphical.target @19.154s
└─multi-user.target @19.154s
└─docker.service @17.865s +1.288s
└─network-online.target @17.863s
└─systemd-networkd-wait-online.service @1.384s +16.478s
└─systemd-networkd.service @972ms +409ms
└─systemd-udevd.service @521ms +442ms
└─systemd-tmpfiles-setup-dev.service @509ms +10ms
└─systemd-sysusers.service @445ms +62ms
└─systemd-remount-fs.service @327ms +110ms
└─systemd-journald.socket @313ms
└─system.slice @306ms
└─-.slice @306msThe log for my previous boot is on this Pastebin, and I used the last because I've since rebooted and started sshd manually (with systemctl start sshd).
Quick edit: the unit file for sshd.service, if that's relevant:
❯ ls -la /etc/systemd/system/multi-user.target.wants/sshd.service
Permissions Size User Group Date Modified Name
lrwxrwxrwx 36 root root 12 Feb 16:50 /etc/systemd/system/multi-user.target.wants/sshd.service -> /usr/lib/systemd/system/sshd.service
❯ cat /usr/lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH Daemon
Wants=sshdgenkeys.service
After=sshdgenkeys.service
After=network.target
[Service]
ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always
[Install]
WantedBy=multi-user.targetLast edited by efficientbug (2020-07-14 17:40:23)
Offline
What is the status of sshd.socket?
Edit:
You can use
systemctl get-defaultTo find the systemd default target. `systemd-analyze critical-chain` already showed it was graphical.target so multi-user.target was reached which should start sshd.service.
Last edited by loqs (2020-07-14 18:24:37)
Offline
What is the status of sshd.socket?
Okay, I believe disabling sshd.socket fixed the issue. Just rebooted and sshd is up and running. Still don't understand how, though. The wiki article has a note about sshd.socket being removed, but it did work when I initially set up ssh server on this computer. From the wiki:
If sshd.socket is enabled when updating to openssh 8.0p1-3, the sshd.socket and sshd@.service units will be copied to /etc/systemd/system/ and reenabled. This is only done to not break existing setups, users are still advised to migrate to sshd.service.
It did break recently. Even attempting to connect multiple times, as I've been doing for the past year or so, did not trigger the socket activation and sshd.socket was enabled until a few hours ago. Is it safe to delete sshd.socket and sshd@.service, then?
I'll be marking this as solved, and maybe this thread will help someone who has set up sshd.socket a while ago. I appreciate the help!
Offline
It is strange that sshd listens on port 22 when it is started automatically, as shown in #1, although you set the port to 31415.
Jul 14 10:57:17 void sshd[11416]: Server listening on 0.0.0.0 port 22.
Jul 14 10:57:17 void sshd[11416]: Server listening on :: port 22.I would start sshd with "-dd" to get more debugging output.
Offline
It is strange that sshd listens on port 22 when it is started automatically, as shown in #1, although you set the port to 31415.
Jul 14 10:57:17 void sshd[11416]: Server listening on 0.0.0.0 port 22. Jul 14 10:57:17 void sshd[11416]: Server listening on :: port 22.I would start sshd with "-dd" to get more debugging output.
Yeah, that was me being dumb when moving my sshd_config to sshd_config.mine, reinstalling sshd to diff the stock file, and forgetting to move it back. But well spotted!
Offline