[Solved] Specifying/Understanding HostKeyAlgorithms for ssh

cfr · 2022-03-02 17:26:58

I'm connecting to a ssh server which has recently been updated. This is triggering

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

The key being offered by the host is ed25519.
The keys for the host in ~/.ssh/known_hosts are ecdsa-sha2-nistp256 ssh-rsa and ssh-ed25519.
However, whereas the offered key is prefixed SHA256, the known ssh-ed25519 key is not.

https://bbs.archlinux.org/viewtopic.php … 5#p1960765 suggests in response to a similar issue

respiranto wrote:

You could also have done:
ssh -o HostKeyAlgorithms=ecdsa-sha2-nistp256,ed25519 -o UpdateHostKeys=yes host
Actually, one may omit setting UpdateHostkeys, as it seems to have been enabled by default in presumably the same version in which HostKeyAlgorithms changed. Possibly a reaction.

Is ed25519 different from ed25519 with the SHA256 prefix?

$ ssh -Q hostkeyalgorithms 
ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ssh-ed25519-cert-v01@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
ssh-rsa-cert-v01@openssh.com
rsa-sha2-256-cert-v01@openssh.com
rsa-sha2-512-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com

This doesn't suggest anything obvious. I was hoping to see something like 'ed25519-sha256' or similar, but either that's not a thing or I just don't know what I'm looking at. If the algorithm (rather than the host key itself?) ed25519 with sha256 really is different from just ed25519, how can I specify the difference in the option I pass to ssh?

I know that I can delete the relevant lines from known_hosts and so trigger a new option to accept the host. But I don't want to do that unless I can be sure nothing nefarious is happening. As I understand it, the ssh command above would let me authenticate the host with the old algorithm and then update known_hosts with a line for the new algorithm. But, assuming ssh-ed25519 is the plain ed25519 one, which algorithm line names the ed25519 with a sha256 prefix?

Last edited by cfr (2022-03-11 13:18:02)

-thc · 2022-03-02 20:08:00

cfr wrote:

However, whereas the offered key is prefixed SHA256, the known ssh-ed25519 key is not.

Can you please show us some messages or output detailing this?

cfr · 2022-03-02 20:52:17

-thc wrote:

Can you please show us some messages or output detailing this?

Output from attempting to connect with ssh <hostname>

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Please contact your system administrator.
Add correct host key in /home/myusername/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/myusername/.ssh/known_hosts:6
Host key for 123.456.789.00 has changed and you have requested strict checking.
Host key verification failed.

Contents of ~/.ssh/known_hosts

123.456.789.4 ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
123.456.789.00 ecdsa-sha2-nistp256 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
aur.archlinux.org,5.9.250.164 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZAVWmj2k+dHTfyum7FyIivGcVUkDFHaXmPNxDwF7l8TvkAN8VDQJHEEGJhALMYtNsQ+kt0gksSh4HZqj9n5hI=
123.456.789.00 ssh-ed25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX dracut-sshd-host-key
123.456.789.00 ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
123.456.789.00 ssh-ed25519 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Note that I apologise if I've obfuscated unnecessarily, but I'm not sure what should and shouldn't be. (I think the IPs should be but the key fingerprints probably didn't need to be. I assume the AUR one is all public.)

Note that early ssh uses a different key, so the SHA256-prefixed key for the server included in known_hosts isn't relevant to the connection attempts I'm concerned with at the moment.

Last edited by cfr (2022-03-02 20:55:22)

loqs · 2022-03-02 20:57:53

ecdsa is Elliptic Curve Digital Signature Algorithm.
ed25519 = Ed25519-SHA-512 is Edwards-curve digital signature algorithm curve 25519 SHA-512

ed25519 with the SHA256 prefix is not listed in the outputs you presented.

Last edited by loqs (2022-03-02 21:35:58)

-thc · 2022-03-02 21:08:27

cfr wrote:

The fingerprint for the ED25519 key sent by the remote host is
SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

This literally means: Here is the SHA256 fingerprint of the changed ED25519 key.

The file known_hosts doesn't contain such fingerprints.

The cause for this message is that the "update" you mentioned at the top of this thread somehow has also lead to the generation of new host keys for this server. The stored (ed25519) key in known_hosts is now different from the one presented by the server.

Last edited by -thc (2022-03-02 21:16:17)

cfr · 2022-03-03 01:47:18

-thc wrote:

This literally means: Here is the SHA256 fingerprint of the changed ED25519 key.

Thanks.

The file known_hosts doesn't contain such fingerprints.

So what is the difference between a line in known_hosts of the form

<ip> ssh-ed25519 SHA256:<key>

and one of the form

<ip> ssh-ed25519 <key>

?

The cause for this message is that the "update" you mentioned at the top of this thread somehow has also lead to the generation of new host keys for this server. The stored (ed25519) key in known_hosts is now different from the one presented by the server.

Under what circumstances is this expected? No new keys were manually generated. Would an upgrade to openssh (or anything else) cause new keys to be generated?

loqs · 2022-03-03 02:29:25

cfr wrote:

Under what circumstances is this expected?

The remote is a different server or it is the same server with a new host key.

cfr wrote:

No new keys were manually generated. Would an upgrade to openssh (or anything else) cause new keys to be generated?

Not if the server is using Arch see https://github.com/archlinux/svntogit-p … ervice#L13
A new host key is only generated if a host key is missing. Something would need to remove the host key then on next service restart it would be regenerated.

cfr · 2022-03-03 04:37:10

loqs wrote:

A new host key is only generated if a host key is missing. Something would need to remove the host key then on next service restart it would be regenerated.

OK, thanks. The server isn't Arch (only the client).

I know that the key files in /etc/ssh on the server aren't new because I checked their dates. I can only think I missed something there or that something else has been changed in the upgrade.

Nobody touches the server but me (except all the people who try to login with random credentials). I did do two major upgrades in quick succession (Fedora 30 to 32; Fedora 32 to 34 a week later.) because the machine lost power during the pandemic and I only recently got physical access to it again. But only config files had recent timestamps in /etc/ssh, so I don't think anything got removed and regenerated.

Anyway, if an upgrade did change something, it certainly isn't an Arch issue.

-thc · 2022-03-03 06:25:02

cfr wrote:

I know that the key files in /etc/ssh on the server aren't new because I checked their dates. I can only think I missed something there or that something else has been changed in the upgrade.
Nobody touches the server but me (except all the people who try to login with random credentials). I did do two major upgrades in quick succession (Fedora 30 to 32; Fedora 32 to 34 a week later.) because the machine lost power during the pandemic and I only recently got physical access to it again. But only config files had recent timestamps in /etc/ssh, so I don't think anything got removed and regenerated.

O.K. - then it's your client. Compare your servers key in "ssh_host_ed25519_key.pub" with the key in the correct entry in your known hosts file

123.456.789.00 ssh-ed25519 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

They should be identical.

cfr wrote:

123.456.789.00 ssh-ed25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX dracut-sshd-host-key

That sailed right past me last time - sorry. I'm pretty sure this line doesn't belong there and may well be the cause of your problem. Make a backup of the file and remove that line.

Last edited by -thc (2022-03-03 06:27:01)

cfr · 2022-03-06 20:31:47

-thc wrote:

O.K. - then it's your client. Compare your servers key in "ssh_host_ed25519_key.pub" with the key in the correct entry in your known hosts file

Thanks. I'll check this on Friday and avoid connecting until then. Maybe I misread a timestamp or something.

cfr wrote:
123.456.789.00 ssh-ed25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX dracut-sshd-host-key
That sailed right past me last time - sorry. I'm pretty sure this line doesn't belong there and may well be the cause of your problem. Make a backup of the file and remove that line.

Aha. Thanks again. I'm going to double-check the host keys against the file first, just to be on the safe side. This line is for early ssh as opposed to regular ssh. It also seems to be the only line for early ssh. But I honestly can't remember the details of how I set all this up. The IP for ssh and early ssh is obviously the same, so I want to double-check what's on the server before removing any safeguards on the client.

cfr · 2022-03-11 13:17:33

I removed the sixth line of ~/.ssh/known_hosts after comparison with the .pub on the server showed it differed.

I still don't know how that erroneous line got into the known_hosts file. The .pub is dated 2016 so the key definitely hasn't changed on the server.

I didn't have to remove the SHA256: line for dracut-ssh.

I'm going to mark this solved because I doubt I'm going to figure out how the erroneous line got in to the client's list of hosts. And ssh now works, at least on the same LAN (but I've no reason to think it won't also work remotely - only the LAN is where I am right now).

Thanks for all the help.

[Now if I could just figure out early ssh on Fedora again ....]

Arch Linux

#1 2022-03-02 17:26:58

[Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#2 2022-03-02 20:08:00

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#3 2022-03-02 20:52:17

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#4 2022-03-02 20:57:53

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#5 2022-03-02 21:08:27

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#6 2022-03-03 01:47:18

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#7 2022-03-03 02:29:25

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#8 2022-03-03 04:37:10

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#9 2022-03-03 06:25:02

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#10 2022-03-06 20:31:47

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

#11 2022-03-11 13:17:33

Re: [Solved] Specifying/Understanding HostKeyAlgorithms for ssh

Board footer