Arch Linux

scott_fakename

Member

Registered: 2012-08-15

Posts: 106

[Solved] Kerberos SSH stopped working, but not kerberized NFS

On my network, I have a Kerberos server, and all the other machines in the network use it for SSH authentication, including my file server.

This has been working fine for months, but recently, my fileserver (and no other machine; all are running arch) has stopped accepting kerberos tickets for SSH authentication. I tried running a debug SSH, and I saw only this that looked relevant:

debug2: input_userauth_request: try method gssapi-with-mic [preauth]
debug3: mm_request_send: entering, type 42 [preauth]
debug3: mm_request_receive_expect: entering, type 43 [preauth]
debug3: mm_request_receive: entering [preauth]
debug3: mm_request_receive: entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive: entering
debug3: monitor_read: checking request 42
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No key table entry found matching host/nas-bot@

stat /etc/krb5.keytab shows that it's got mode 600 and is owned by root:root; the machine's /etc/hostname is correct (or, it hasn't changed from before, it's the machine's FQDN).

Another machine I own continues to accept kerberos tickets for SSH. It too has /etc/krb5.keytab owned by root and with mode 600; it's /etc/hostname is also the machine's FQDN.

I don't think kerberos is broken generally on that fileserver, because I also use kerberos to secure my NFS server; kerberized NFS still works.

The /etc/ssh/sshd_config file still shows these lines (on all machines):

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes

I tried re-creating the keytab, but SSH is till not accepting kerberos tickets.

My best guess is that for some reason, SSH on that machine is resolving its own hostname as just "nas-bot" instead of the FQDN, but I can't think of any reason that SSH, and not NFS, would be behaving this way.

Can anyone think of a reason why that might be, or other things to check?

Last edited by scott_fakename (2023-03-05 00:22:33)

scott_fakename

Member

Registered: 2012-08-15

Posts: 106

Re: [Solved] Kerberos SSH stopped working, but not kerberized NFS

I noticed that it started doing this shortly after I installed Squid, so I tried looking at all the packages that had a ctime from within the last two days and comparing them to the dependencies of squid, from here: https://archlinux.org/packages/community/x86_64/squid/

I removed all of them that I could, and now it's back working again.

I'm not sure if it was a transient thing, or if somehow squid or one of its dependencies caused it, or if it just happens to be working now temporarily.

scott_fakename

Member

Registered: 2012-08-15

Posts: 106

Re: [Solved] Kerberos SSH stopped working, but not kerberized NFS

As an update, it wasn't automatically fixed by uninstalling those things; it's busted again today unfortunately. It seems flaky, but unclear how to debug.

scott_fakename

Member

Registered: 2012-08-15

Posts: 106

Re: [Solved] Kerberos SSH stopped working, but not kerberized NFS

It seems to be flakily using different host names between reboots, for some reason.

On the troublesome system, if I watch sshd with -d sometimes it will try to get a ticket from the kdm under the host's fully qualified domain, and sometimes it will only use the hostname itself.

from the troubled system:

$ ssh -vvv eris-bot
# snip
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Server host/eris-bot@URAXA.LAN not found in Kerberos database


debug3: send packet: type 50

It should be using eris-bot.uraxa.lan as the host.

On eris-bot, the machine i'm trying to SSH to from the bad host:

(0) 10:16:58 [scott@eris-bot ~]$ getent hosts eris-bot
192.168.0.206   eris-bot.uraxa.lan eris-bot

On the bad host:

(0) 10:17:21 [scott@nas-bot ~]$ getent hosts eris-bot
192.168.0.206   eris-bot
(0) 10:18:19 [scott@nas-bot ~]$ getent hosts eris-bot.uraxa.lan
192.168.0.206   eris-bot.uraxa.lan

On a different host that works:

(0) 10:18:38 [scott@neptune-bot ~]$ getent hosts eris-bot
192.168.0.206   eris-bot.uraxa.lan eris-bot

There's something screwy with DNS on the troublesome host.

scott_fakename

Member

Registered: 2012-08-15

Posts: 106

Re: [Solved] Kerberos SSH stopped working, but not kerberized NFS

The troubled host was running systemd-resolved, while no other hosts were. I disabled it, and it started working. All my hosts are set up in /etc/hosts so I don't need it.

I'll mark as fixed if it doesn't happen in the next day or so.

Last edited by scott_fakename (2023-03-04 18:29:50)