Arch Linux

probackup-nl

Member

From: Delft

Registered: 2017-11-15

Posts: 92

Website

Feb 15th upgrade breaks dropbear ssh publickey RSA login

Working remotely using Mac OS openssl 8.0 ssh client (with publickeys) on the client and dropbear on the Arch server.

Pacman -Syu broke publickey login. Dissecting the issue leads to Feb 15th updates.

Trying to ssh as root user with a  publickey, after Feb 15th updates, fails with:

dropbear[1104410]: Login attempt for nonexistent user

Which of these (15) packages is the most likely candidate to have broken dropbear root login?

binutils-2.38-3  elfutils-0.186-5  gcc-libs-11.2.0-3  glibc-2.35-2  libelf-0.186-5  libsysprof-capture-3.42.1-3  libtool-2.4.6+59+gb55b1cc8-2  linux-5.16.9.arch1-1
              linux-api-headers-5.16.8-1  linux-firmware-20220209.6342082-1  linux-firmware-whence-20220209.6342082-1  mpfr-4.1.0.p13-2  nano-6.1-1  util-linux-2.37.4-1
              util-linux-libs-2.37.4-1

Reverting (downgrading) back to 2022 Feb 14th https://archive.archlinux.org/repos/202 … o/os/$arch fixes the login.

Diffing the -vvv output shows as the last identical lines:

debug1: Offering public key: /Users/xxx/.ssh/id_rsa RSA SHA256:... explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply

The succeeding ssh login then continues with:

debug3: receive packet: type 60
debug1: Server accepts key: /Users/xxx/.ssh/id_rsa RSA SHA256:... explicit agent

The failing login instead continues with:

debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /Users/xxx/.ssh/id_ed25519 ED25519 SHA256:... explicit agent

Workaround
After manually restarting dropbear, publickey login works as before.

Some trigger should be improved so that pacman automagically restarts dropbear.

Last edited by probackup-nl (2022-02-20 16:23:37)

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 14,852

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

Something that stood out to me in your post :
working login appears to offer an RSA key while the failing one offers ED25519

dropbear uses libxcrypt which was updated recently, but the february 7th repo does have the latest version of it.

No idea which it could be, you may have to step one day at a time using the archive to narrow down the possible culprits.

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

paulkerry

Member

From: Sheffield, UK

Registered: 2014-10-02

Posts: 611

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

I don't use dropbear so can't comment on your specific issue, but could this be libnss related changes in glibc?
I notice from a recent update that /usr/lib/libnss_files.so and /usr/lib/libnss_files-2.33.so have now been removed from Arch, leaving just /usr/lib/libnss_files.so.2
In the past, all of these files have been needed for things like ssh chroot jails and if they were missing, would lead to "You don't exist, go away!" types of messages.

Regards
Paul

Last edited by paulkerry (2022-02-19 21:21:18)

probackup-nl

Member

From: Delft

Registered: 2017-11-15

Posts: 92

Website

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

I don't use dropbear so can't comment on your specific issue, but could this be libnss related changes in glibc?
I notice from a recent update that /usr/lib/libnss_files.so and /usr/lib/libnss_files-2.33.so have now been removed from Arch, leaving just /usr/lib/libnss_files.so.2
In the past, all of these files have been needed for things like ssh chroot jails and if they were missing, would lead to "You don't exist, go away!" types of messages.

I narrowed the issue down to Feb 15th updates.

With Feb 14th. state:

# ls -la /usr/lib/libnss_files.so
lrwxrwxrwx 1 root root 17 May 12  2021 /usr/lib/libnss_files.so -> libnss_files.so.2
# ls -la  /usr/lib/libnss_files-2.33.so
-rwxr-xr-x 1 root root 51376 May 12  2021 /usr/lib/libnss_files-2.33.so

With Feb 15th. state:

ls: cannot access '/usr/lib/libnss_files-2.33.so': No such file or directory
ls: cannot access '/usr/lib/libnss_files.so': No such file or directory

Another server has the same issue.

Downgrading to util-linux-libs-2.37.3-2 and restarting dropbear solves the issue.

Last edited by probackup-nl (2022-02-20 16:24:21)

paulkerry

Member

From: Sheffield, UK

Registered: 2014-10-02

Posts: 611

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

So did you try using the previous glibc package that contains those files?

probackup-nl

Member

From: Delft

Registered: 2017-11-15

Posts: 92

Website

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

So did you try using the previous glibc package that contains those files?

I did need to restart dropbear.

systemctl restart dropbear

is the only thing I have done.

paulkerry

Member

From: Sheffield, UK

Registered: 2014-10-02

Posts: 611

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

That doesn't really answer my question, unless I am misunderstanding you which is highly likely.
In your initial post, you asked which package upgrade might have caused dropbear ssh logins to stop working from a list of 15x packages that were upgraded, and I'm suggesting that it might be the glibc package update, so you could try installing using a locally cached version of the older glibc package or downloading that package from the ALA - https://wiki.archlinux.org/title/Arch_Linux_Archive

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,654

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

downgrading glibc alone is a bad idea, it's likely much of the system will stop working, possibly including pacman.

probackup-nl

Member

From: Delft

Registered: 2017-11-15

Posts: 92

Website

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

Thank you for trying to help me to find the root cause for this issue.

When using the archive, the other 15 packages are downgraded too. (So I think I can't pinpoint the issue to glibc).

When downgrading only glibc using:

# pacman -U /var/cache/pacman/pkg/glibc-2.33-5-x86_64.pkg.tar.zst
loading packages...
warning: downgrading package glibc (2.35-2 => 2.33-5)

error messages occur, like:

systemd-detect-virt: /usr/lib/libc.so.6: version `GLIBC_2.34' not found (required by /usr/lib/libgcc_s.so.1)

though I can still login using dropbear.

probackup-nl

Member

From: Delft

Registered: 2017-11-15

Posts: 92

Website

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

downgrading glibc alone is a bad idea, it's likely much of the system will stop working, possibly including pacman.

Thank you for your rapid warning.

I was very lucky. My system didn't stop responding (this time) after downgrading only glibc. Immediately upgraded it again.

probackup-nl

Member

From: Delft

Registered: 2017-11-15

Posts: 92

Website

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

After Feb 15th upgrade:

# lsof -p 1209191 | grep /usr/lib/
dropbear 1209191 root  DEL    REG               0,24          2066169 /usr/lib/libc.so.6
dropbear 1209191 root  mem    REG               0,24          1973351 /usr/lib/libcrypt.so.2.0.0 (path dev=0,25)
dropbear 1209191 root  mem    REG               0,24          1950532 /usr/lib/libz.so.1.2.11 (path dev=0,25)
dropbear 1209191 root  DEL    REG               0,24          2066205 /usr/lib/libutil.so.1
dropbear 1209191 root  DEL    REG               0,24          2066160 /usr/lib/ld-linux-x86-64.so.2
# cd /usr/lib;pacman -Qo libc.so.6 libcrypt.so.2.0.0 libz.so.1.2.11 libutil.so.1 ld-linux-x86-64.so.2
/usr/lib/libc.so.6 is owned by glibc 2.35-2
/usr/lib/libcrypt.so.2.0.0 is owned by libxcrypt 4.4.28-1
/usr/lib/libz.so.1.2.11 is owned by zlib 1:1.2.11-5
/usr/lib/libutil.so.1 is owned by glibc 2.35-2
/usr/lib/ld-linux-x86-64.so.2 is owned by glibc 2.35-2

that leaves glibc as the only possible candidate for breaking dropbear (or not?).

paulkerry

Member

From: Sheffield, UK

Registered: 2014-10-02

Posts: 611

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

Sorry guys - you are correct of course Skimmia.

probackup-nl

Member

From: Delft

Registered: 2017-11-15

Posts: 92

Website

Re: Feb 15th upgrade breaks dropbear ssh publickey RSA login

Could the archlinux-contrib script checkservices script be a solution to trigger a message that the dropbear systemd service should be restarted on such glibc upgrades?