[SOLVED] Unable to login after pacman -Syu

sumebrius · 2021-03-03 08:46:27

So this is an odd issue, that has me stumped. After running pacman -Syu recently, it looks like my system is completely unable to do anything involving user or group lookups. Unfortunately this involves logging in, and most services, so currently completely unusable...

First sign of the problem was during the pacman transaction, with a swathe of "Failed to resolve user" errors in the 30-systemd-tmpfiles hook, eg:

[ALPM-SCRIPTLET] /usr/lib/tmpfiles.d/bind.conf:1: Failed to resolve user 'named': No such process

Immediately after the transactions, sudo commands failed before even providing a password prompt. SSH attempts were immediately terminated.

After shutting down, getting it plugged into a keyboard & monitor (usually it's headless), and booting, the system booted albeit with most services failing. I can get to a login prompt, but as soon as entering a username, sh appears to just restart, and I'm booted back to the login prompt.

I am able to boot the system from archiso - checking the journals, it looks like any attempt at a user lookup is resulting in a coredump...

                                       Stack trace of thread 828:
                                       #0  0x00007f059ddc2c18 __nss_module_get_function (libc.so.6 + 0x122c18)
                                       #1  0x00007f059ddc0c32 __nss_lookup (libc.so.6 + 0x120c32)
                                       #2  0x00007f059dd6b691 getpwnam_r@@GLIBC_2.2.5 (libc.so.6 + 0xcb691)
                                       #3  0x00007f059dd6b05c getpwnam (libc.so.6 + 0xcb05c)
                                       #4  0x00007f059e0cd141 get_user_creds (libsystemd-shared-247.so + 0x14d141)
                                       #5  0x000055d4080e9891 n/a (systemd + 0xe9891)
                                       #6  0x000055d408056723 n/a (systemd + 0x56723)
                                       #7  0x000055d40808a086 n/a (systemd + 0x8a086)
                                       #8  0x000055d40808c3aa n/a (systemd + 0x8c3aa)
                                       #9  0x000055d40808c890 n/a (systemd + 0x8c890)
                                       #10 0x000055d40805ff5b n/a (systemd + 0x5ff5b)
                                       #11 0x000055d40807103a n/a (systemd + 0x7103a)
                                       #12 0x00007f059e13581a n/a (libsystemd-shared-247.so + 0x1b581a)
                                       #13 0x00007f059e135cab sd_event_dispatch (libsystemd-shared-247.so + 0x1b5cab)
                                       #14 0x00007f059e137061 sd_event_run (libsystemd-shared-247.so + 0x1b7061)
                                       #15 0x000055d408076ea3 n/a (systemd + 0x76ea3)
                                       #16 0x000055d408037208 n/a (systemd + 0x37208)
                                       #17 0x00007f059dcc7b25 __libc_start_main (libc.so.6 + 0x27b25)
                                       #18 0x000055d40803840e n/a (systemd + 0x3840e)

The issue doesn't appear to be hardware or disk failure related. No issues running off archiso, and the SDD reports no SMART issues, and no obvious file corruption in anything looked at.

I used pacstrap to re-install (from repo, not cache) anything that looked remotely relevant from the original pacman transaction, but unfortunately no difference:
glibc gcc-libs libcap libuv linux linux-headers glib2 sssd

I'm guessing from the coredump that the issue appears to be in the libc.so.6 library, but I'm at a bit of a loss as the next steps in troubleshooting - does anyone have any ideas?

Relevant logs:
Pacman logs of the initial transaction: https://pastebin.com/YBWSyRzG
journalctl logs of the latest boot of the system: https://pastebin.com/UqTz77BX

Last edited by sumebrius (2021-03-04 09:04:20)

seth · 2021-03-03 13:32:25

Just a hunch, but disable the ufw.service.

Also post your /etc/nsswitch.conf, /etc/group and /etc/passwd and ensure that systemd-libs is up to date and not broken (pacman -Qkk systemd-libs)

sumebrius · 2021-03-04 09:02:47

Thanks seth.

It was my nsswitch.conf - I had completely missed a pacnew version of it from around a month earlier, and my old version was, to be frank, a hot mess. From memory the result of a bit of a bodge a few years ago.

After overwriting nsswitch.conf with the pacnew version, everything's back to normal!

THANK YOU!

Arch Linux

#1 2021-03-03 08:46:27

[SOLVED] Unable to login after pacman -Syu

#2 2021-03-03 13:32:25

Re: [SOLVED] Unable to login after pacman -Syu

#3 2021-03-04 09:02:47

Re: [SOLVED] Unable to login after pacman -Syu

Board footer