You are not logged in.
Pages: 1
I recently updated my system and discovered I cannot authenticate to root anymore via tty or ssh. I was able to get back into the system with dropbear, which led me to believe it was a PAM issue. After disabling UsePAM in OpenSSH authentication works again. However, tools that rely on PAM are still failing and I'd like to figure out why password auth does not work anymore. I would like to find out what happeend and fix this.
Before the system update:
Jul 25 06:27:47 mainsys kernel: audit: type=1110 audit(1627219667.888:4164): pid=1461724 uid=1000 auid=1000 ses=13 msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
Jul 25 06:27:47 mainsys kernel: audit: type=1105 audit(1627219667.888:4165): pid=1461724 uid=1000 auid=1000 ses=13 msg='op=PAM:session_open grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
... short time later ...
Jul 25 07:24:02 mainsys systemd[1]: shadow.service: Main process exited, code=exited, status=1/FAILURE
Jul 25 07:24:02 mainsys systemd[1]: shadow.service: Failed with result 'exit-code'.
Jul 25 07:24:02 mainsys kernel: audit: type=1131 audit(1627223042.881:4257): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=shadow comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? a
ddr=? terminal=? res=failed'
Jul 25 07:24:16 mainsys systemd[1]: Started Verify integrity of password and group files.
Jul 25 07:24:16 mainsys kernel: audit: type=1130 audit(1627223056.344:4258): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=shadow comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? a
ddr=? terminal=? res=success'
Jul 25 07:24:16 mainsys systemd[1]: shadow.service: Deactivated successfully.
Jul 25 07:24:16 mainsys kernel: audit: type=1131 audit(1627223056.374:4259): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=shadow comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? a
ddr=? terminal=? res=success'
... do an update ...
Jul 25 07:26:16 mainsys pacman[1482262]: ==> WARNING: Possibly missing firmware for module: xhci_pci
Jul 25 07:26:16 mainsys pacman[1482262]: -> Running build hook: [filesystems]
Jul 25 07:26:16 mainsys pacman[1482262]: -> Running build hook: [keyboard]
Jul 25 07:26:16 mainsys pacman[1482262]: -> Running build hook: [fsck]
Jul 25 07:26:16 mainsys pacman[1482262]: ==> Generating module dependencies
Jul 25 07:26:17 mainsys pacman[1482262]: ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
Jul 25 07:26:17 mainsys pacman[1482262]: ==> Image generation successful
Jul 25 07:26:17 mainsys pacman[1482262]: ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
Jul 25 07:26:17 mainsys pacman[1482262]: -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
Jul 25 07:26:17 mainsys pacman[1482262]: ==> Starting build: 5.13.4-arch2-1
Jul 25 07:26:17 mainsys pacman[1482262]: -> Running build hook: [base]
Jul 25 07:26:17 mainsys pacman[1482262]: -> Running build hook: [udev]
Jul 25 07:26:17 mainsys pacman[1482262]: -> Running build hook: [modconf]
Jul 25 07:26:17 mainsys pacman[1482262]: -> Running build hook: [block]
Jul 25 07:26:17 mainsys pacman[1482262]: ==> WARNING: Possibly missing firmware for module: wd719x
Jul 25 07:26:17 mainsys pacman[1482262]: ==> WARNING: Possibly missing firmware for module: aic94xx
Jul 25 07:26:18 mainsys pacman[1482262]: ==> WARNING: Possibly missing firmware for module: xhci_pci
Jul 25 07:26:18 mainsys pacman[1482262]: -> Running build hook: [filesystems]
Jul 25 07:26:19 mainsys pacman[1482262]: -> Running build hook: [keyboard]
Jul 25 07:26:19 mainsys pacman[1482262]: -> Running build hook: [fsck]
Jul 25 07:26:19 mainsys pacman[1482262]: ==> Generating module dependencies
Jul 25 07:26:20 mainsys pacman[1482262]: ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fallback.img
Jul 25 07:26:20 mainsys pacman[1482262]: ==> Image generation successful
Jul 25 07:26:20 mainsys pacman[1482262]: running '90-packagekit-refresh.hook'...
Jul 25 07:27:11 mainsys libvirtd[2496]: argument unsupported: QEMU guest agent is not configured
Jul 25 07:29:18 mainsys kernel: audit: type=1100 audit(1627223357.982:4363): pid=1484740 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/sshd" hostname=172.24.40.36 addr=172.24.40.36 terminal=ssh res=failed'
Jul 25 07:29:22 mainsys kernel: audit: type=1100 audit(1627223362.945:4364): pid=1484740 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/sshd" hostname=172.24.40.36 addr=172.24.40.36 terminal=ssh res=failed'
Jul 25 07:29:27 mainsys kernel: audit: type=2100 audit(1627223367.944:4365): pid=1484740 uid=0 auid=4294967295 ses=4294967295 msg='pam_faillock uid=0 exe="/usr/bin/sshd" hostname=? addr=? terminal=? res=success'
Jul 25 07:29:27 mainsys kernel: audit: type=1100 audit(1627223367.944:4366): pid=1484740 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/sshd" hostname=172.24.40.36 addr=172.24.40.36 terminal=ssh res=failed'
Jul 25 07:29:34 mainsys kernel: audit: type=1100 audit(1627223374.329:4367): pid=1484744 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/sshd" hostname=172.24.40.36 addr=172.24.40.36 terminal=ssh res=failed'
Jul 25 07:29:49 mainsys kernel: audit: type=1100 audit(1627223389.358:4368): pid=1484754 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/sshd" hostname=172.24.40.36 addr=172.24.40.36 terminal=ssh res=failed'I attempted to unlock the root account just in case:
passwd -u root; faillock --user root --reset
passwd: password expiry information changed.But that made no difference. I also commented out faillock in pam to see if that would help but again, no difference.
Example failed log line:
Jul 25 09:22:58 mainsys audit[19828]: USER_AUTH pid=19828 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/sshd" hostname=172.24.42.73 addr=172.24.42.73 terminal=ssh res=failed'
Jul 25 09:22:58 mainsys kernel: audit: type=1100 audit(1627230178.628:487): pid=19828 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/sshd" hostname=172.24.42.73 addr=172.24.42.73 terminal=ssh res=failed'
Jul 25 09:23:00 mainsys sshd[19828]: Failed password for root from 172.24.42.73 port 59912 ssh2The following versions are installed:
pam 1.5.1-1
pambase 20210605-2
shadow 4.8.1-4
systemd-249.2-1
(systemd 249.2-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2
-IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified))
Linux 5.10.52-1-lts #1 SMP Tue, 20 Jul 2021 16:46:09 +0000 x86_64 GNU/LinuxContents of my pam.d directory:
$ls -la /etc/pam.d
total 140
drwxr-xr-x 2 root root 4096 Jul 25 08:41 .
drwxr-xr-x 73 root root 4096 Jul 25 08:37 ..
-rw-r--r-- 1 root root 160 Sep 7 2020 chage
-rw-r--r-- 1 root root 234 Jul 22 11:25 chfn
-rw-r--r-- 1 root root 103 Sep 7 2020 chgpasswd
-rw-r--r-- 1 root root 174 Sep 7 2020 chpasswd
-rw-r--r-- 1 root root 234 Jul 22 11:25 chsh
-rw-r--r-- 1 root root 170 Jul 24 05:16 cockpit
-rw-r--r-- 1 root root 160 Sep 7 2020 groupadd
-rw-r--r-- 1 root root 160 Sep 7 2020 groupdel
-rw-r--r-- 1 root root 103 Sep 7 2020 groupmems
-rw-r--r-- 1 root root 160 Sep 7 2020 groupmod
-rw-r--r-- 1 root root 263 Jul 22 11:25 login
-rw-r--r-- 1 root root 174 Sep 7 2020 newusers
-rw-r--r-- 1 root root 274 Jun 7 23:54 other
-rw-r--r-- 1 root root 198 Sep 7 2020 passwd
-rw-r--r-- 1 root root 155 Jun 3 07:37 polkit-1
-rw-r--r-- 1 root root 86 Jul 22 11:25 runuser
-rw-r--r-- 1 root root 86 Jul 22 11:25 runuser-l
-rw-r--r-- 1 root root 27 Mar 4 08:59 screen
-rw-r--r-- 1 root root 160 Sep 7 2020 shadow
-rw-r--r-- 1 root root 232 Apr 19 04:32 sshd
-rw-r--r-- 1 root root 500 Jul 22 11:25 su
-rw-r--r-- 1 root root 97 Jun 11 23:10 sudo
-rw-r--r-- 1 root root 500 Jul 22 11:25 su-l
-rw-r--r-- 1 root root 1297 Jul 25 08:02 system-auth
-rw-r--r-- 1 root root 131 Jul 23 13:21 systemd-user
-rw-r--r-- 1 root root 143 Jun 7 23:54 system-local-login
-rw-r--r-- 1 root root 627 Jul 25 08:02 system-login
-rw-r--r-- 1 root root 143 Jun 7 23:54 system-remote-login
-rw-r--r-- 1 root root 260 Jun 7 23:54 system-services
-rw-r--r-- 1 root root 160 Sep 7 2020 useradd
-rw-r--r-- 1 root root 160 Sep 7 2020 userdel
-rw-r--r-- 1 root root 160 Sep 7 2020 usermod
-rw-r--r-- 1 root root 124 Feb 9 22:48 vlockContents of my system-auth pam file:
#%PAM-1.0
auth required pam_shells.so
auth requisite pam_nologin.so
auth include system-auth
account required pam_access.so
account required pam_nologin.so
account include system-auth
password include system-auth
session optional pam_loginuid.so
session optional pam_keyinit.so force revoke
session include system-auth
session optional pam_motd.so motd=/etc/motd
session optional pam_mail.so dir=/var/spool/mail standard quiet
-session optional pam_systemd.so
session required pam_env.so debug user_readenv=0Please note I added the debug on pam env to try to troubleshoot.
Contents of my system-auth file:
#%PAM-1.0
#auth required pam_faillock.so preauth
# Optionally use requisite above if you do not want to prompt for the password
# on locked accounts.
auth [success=2 default=ignore] pam_unix.so try_first_pass nullok
-auth [success=1 default=ignore] pam_systemd_home.so
auth [default=die] pam_faillock.so authfail
auth optional pam_permit.so
auth required pam_env.so debug user_readenv=0
auth required pam_faillock.so authsucc
# If you drop the above call to pam_faillock.so the lock will be done also
# on non-consecutive authentication failures.
-account [success=1 default=ignore] pam_systemd_home.so
account required pam_unix.so
account optional pam_permit.so
account required pam_time.so
-password [success=1 default=ignore] pam_systemd_home.so
password required pam_unix.so try_first_pass nullok shadow sha512
password optional pam_permit.so
session required pam_limits.so
session required pam_unix.so
session optional pam_permit.soOffline
Pages: 1