You are not logged in.

#1 2020-12-06 21:53:12

MountainX
Member
Registered: 2016-02-08
Posts: 371

sudo password not accepted until after I reboot

I'm running Arch on a device that I access remotely via SSH with keypair authentication (no password required for SSH login as my user). SSH does not permit root login.

The issue is that after I am logged in via SSH, my password for sudo is rejected three times. My password is correct.

After a reboot, all is good -- at least for a while. I have seen this happen a few times in recent weeks. I have no idea what causes it.

$ sudo -s
[sudo] password for myuser: 
Sorry, try again.
[sudo] password for myuser: 
Sorry, try again.
[sudo] password for myuser: 
sudo: 3 incorrect password attempts

This is the output of journalctl -r immediately after executing the above command:

Dec 06 16:19:39 archie sudo[2227186]: myuser : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/home/myuser ; USER=root ; COMMAND=/bin/bash
Dec 06 16:19:37 archie kernel: audit: type=1100 audit(1607289577.448:9102): pid=2227186 uid=1000 auid=1000 ses=859 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=failed'
Dec 06 16:19:37 archie audit[2227186]: USER_AUTH pid=2227186 uid=1000 auid=1000 ses=859 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=failed'
Dec 06 16:19:33 archie kernel: audit: type=1100 audit(1607289573.091:9101): pid=2227186 uid=1000 auid=1000 ses=859 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=failed'
Dec 06 16:19:33 archie audit[2227186]: USER_AUTH pid=2227186 uid=1000 auid=1000 ses=859 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=failed'
Dec 06 16:19:28 archie kernel: audit: type=1100 audit(1607289568.545:9100): pid=2227186 uid=1000 auid=1000 ses=859 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=failed'
Dec 06 16:19:28 archie audit[2227186]: USER_AUTH pid=2227186 uid=1000 auid=1000 ses=859 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=failed'

After rebooting, it works (with the same password, which was correct all along):

$ sudo -s
[sudo] password for myuser:
root@archie [/home/myuser]
#

There are no unprocessed .pacnew files on this system (and it is fully updated). In particular, the last  "/etc/pam.d/login.pacnew" was processed (I replaced the old file with the new).

# cat /etc/pam.d/login
#%PAM-1.0

auth       required     pam_securetty.so
auth       requisite    pam_nologin.so
auth       include      system-local-login
account    include      system-local-login
session    include      system-local-login

# cat /etc/pam.d/sudo
#%PAM-1.0
auth            include         system-auth
account         include         system-auth
session         include         system-auth


# cat /etc/pam.d/system-auth
#%PAM-1.0

auth       required                    pam_faillock.so      preauth
# Optionally use requisite above if you do not want to prompt for the password
# on locked accounts.
auth       [success=2 default=ignore]  pam_unix.so          try_first_pass nullok
-auth      [success=1 default=ignore]  pam_systemd_home.so
auth       [default=die]               pam_faillock.so      authfail
auth       optional                    pam_permit.so
auth       required                    pam_env.so
auth       required                    pam_faillock.so      authsucc
# If you drop the above call to pam_faillock.so the lock will be done also
# on non-consecutive authentication failures.

-account   [success=1 default=ignore]  pam_systemd_home.so
account    required                    pam_unix.so
account    optional                    pam_permit.so
account    required                    pam_time.so

-password  [success=1 default=ignore]  pam_systemd_home.so
password   required                    pam_unix.so          try_first_pass nullok shadow
password   optional                    pam_permit.so

session    required                    pam_limits.so
session    required                    pam_unix.so
session    optional                    pam_permit.so

I have checked these bug reports and those don't appear related to my issue.

Apparent bug in faillock: https://bugs.archlinux.org/task/67644
Removed tally module referenced: https://bugs.archlinux.org/task/67641
.pam_environment parsing: https://bugs.archlinux.org/task/67636

Last edited by MountainX (2020-12-06 21:57:54)

Offline

#2 2020-12-06 21:55:11

GaKu999
Member
From: US/Eastern
Registered: 2020-06-21
Posts: 696

Re: sudo password not accepted until after I reboot

Input method derp?
And reboot gave you your standard input method?

Like keyboard mapping?
Did you reboot the SSH server or your local client?

You'll need to be able to replicate this with steps, otherwise I'd blame cosmic rays.

Last edited by GaKu999 (2020-12-06 21:56:56)


My reposSome snippets

Heisenberg might have been here.

Offline

#3 2020-12-06 22:03:25

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

GaKu999 wrote:

Input method derp?
And reboot gave you your standard input method?

Not sure what derp means. Sorry.
As far as input method, it is not related to any change in keyboard mapping or anything like that. I'm sure of that. Other than the password for sudo, I can work on the remote device and all text typed is exactly as expected.

GaKu999 wrote:

Did you reboot the SSH server or your local client?

It is the remote device that has to be rebooted. The problem is strictly with that remote device. It has nothing to do with my local client.

GaKu999 wrote:

You'll need to be able to replicate this with steps, otherwise I'd blame cosmic rays.

That's the problem. When it happened about a month ago I checked everything and could not find any issues, so I put it down to cosmic rays. But when it happened two more times, I can no longer blame those rays. But I cannot reproduce it in normal circumstances. After the reboot, everything is working fine, and it will probably continue to work fine for a week or more.

Last edited by MountainX (2020-12-06 22:16:49)

Offline

#4 2020-12-06 22:25:55

GaKu999
Member
From: US/Eastern
Registered: 2020-06-21
Posts: 696

Re: sudo password not accepted until after I reboot

So, after an uncertain amount of uptime standard auth fails to no avail...
And this is a remote SSH server that can (teorically) have quite a lot of uptime...

...

Anything funny on the logs?
How many brute force SSH attempts do you get /month?

Can you physically access the server?


My reposSome snippets

Heisenberg might have been here.

Offline

#5 2020-12-06 22:34:39

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

GaKu999 wrote:

So, after an uncertain amount of uptime standard auth fails to no avail...
And this is a remote SSH server that can (theoretically) have quite a lot of uptime...

Yes to both.

GaKu999 wrote:

Anything funny on the logs?
How many brute force SSH attempts do you get /month?

Nothing funny in the logs, and the remote device doesn't get any brute force SSH attempts because it is behind a good firewall.

GaKu999 wrote:

Can you physically access the server?

Not at this moment. Today I had to call for assistance to get someone to physically reboot it. I did have the server here about a month ago and I saw the issue for the first time then. It doesn't appear to be related to SSH because at that time it happened the same way when I was working on the device via a locally connected console (monitor, keyboard, etc.).

Offline

#6 2020-12-06 23:30:42

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: sudo password not accepted until after I reboot

When it next happens check if faillock has been triggered by three bad passwords with:

faillock

Offline

#7 2020-12-06 23:41:40

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

loqs wrote:

When it next happens check if faillock has been triggered by three bad passwords with:

faillock

Are you wondering if faillock was already in effect, before I attempted to login as in my OP? I can definitely say that this was not the case today or in the earlier episodes.

However, after I tried 3 times earlier today, there is a faillock file for my user. It has these permissions and a timestamp that matches my attempts.

-rw-rw----  1 myuser myuser  64 Dec  6 16:42 myuser

Here's what the faillock command returns as of now:

myuser:
When                Type  Source                                           Valid
2020-12-06 16:42:57 TTY                                                        V

Login / password are working fine currently.

Offline

#8 2020-12-06 23:56:02

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: sudo password not accepted until after I reboot

faillock having an entry means it has recorded an invalid login attempt.  You can clear it with:

faillock --reset

It should also automatically be cleared by the next valid login for that username.

Offline

#9 2020-12-06 23:59:36

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

loqs wrote:

faillock having an entry means it has recorded an invalid login attempt.  You can clear it with:

faillock --reset

It should also automatically be cleared by the next valid login for that username.

Yes, as I mentioned in the OP, even with the correct password, the login attempt was recorded as invalid, until after I rebooted.

Also, it appears that when you login via SSH using a keypair, that entry is not automatically cleared. I manually cleared it after login.

Offline

#10 2020-12-07 00:02:26

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: sudo password not accepted until after I reboot

If every ssh login adds an entry to faillock then three logins later are you locked out?

Offline

#11 2020-12-07 00:07:37

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

loqs wrote:

If every ssh login adds an entry to faillock then three logins later are you locked out?

Sorry, I don't understand.

Last edited by MountainX (2020-12-07 00:08:12)

Offline

#12 2020-12-07 00:11:26

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: sudo password not accepted until after I reboot

faillock using the default /etc/security/faillock.conf will reject login attempts for 10 minutes after three failed attempts within 15 minutes.
If every login attempt is recorded by faillock as invalid then I was wondering after three logins if faillock was rejecting further logins.
Edit:
As a work around:
deny = 0
in /etc/security/faillock.conf  will stop faillock rejecting login attempts.

Last edited by loqs (2020-12-07 00:19:04)

Offline

#13 2020-12-07 00:22:06

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

loqs wrote:

faillock using the default /etc/security/faillock.conf will reject login attempts for 10 minutes after three failed attempts within 15 minutes.
If every login attempt is recorded by faillock as invalid then I was wondering after three logins if faillock was rejecting further logins.

Do you have any suggestions of what I could look for in the logs to help answer your question?

My very first login attempt of today was rejected. There was no prior attempt within faillock's window. All subsequent logins were also rejected -- until I rebooted. I am having trouble finding anything incriminating in the journal.

After rebooting, no login attempts are rejected (at least until this issue pops up again, maybe a week or two later).

Even when I cannot login, I can still access the device via SSH. I just cannot sudo because my (correct) password is rejected.

An SSH login using a keypair apparently doesn't clear the faillock entry (nor does it create one).

Offline

#14 2020-12-07 00:42:15

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: sudo password not accepted until after I reboot

With the keypair login I suspect sshd does not use pam for authentication only for account and session modules which explains why you can still login as the faillock check is in the auth module.

I do not have any immediate suggestions on how to debug the issue.

Offline

#15 2020-12-07 01:22:44

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

I did just find something potentially interesting in the journal. It does not appear to be directly related to my sudo login attempt. But maybe it is indirectly related?

The most interesting line to me is:

pam_unix(sudo:auth): auth could not identify password for [myuser]

What I see below seems related to rsync. There is an rsync process running on another server that logs in to this device via SSH with my user account (and a keypair) and then utilizes this option: --rsync-path="sudo rsync". I intended to have "myuser ALL=(ALL) NOPASSWD: /usr/bin/rsync" in a file in /etc/sudoers.d/, but I had left that out by mistake (until now).

After the discussion here, I'm wondering if the "sudo rsync" command is triggering faillock. I already fixed the missing sudo authorization line for rsync.

Here's more of the journal. I don't know exactly what some of these journal entries mean.

Dec 06 16:12:18 archie sudo[2225653]: pam_systemd_home(sudo:auth): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Dec 06 16:12:18 archie dbus-daemon[440]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Dec 06 16:12:18 archie dbus-daemon[440]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.10356' (uid=0 pid=2225653 comm="sudo rsync --server >
Dec 06 16:12:18 archie sudo[2225653]: pam_unix(sudo:auth): auth could not identify password for [myuser]
Dec 06 16:12:18 archie audit[2225653]: USER_AUTH pid=2225653 uid=1000 auid=1000 ses=861 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=failed'
Dec 06 16:12:18 archie audit[2225653]: RESP_ACCT_UNLOCK_TIMED pid=2225653 uid=1000 auid=1000 ses=861 msg='pam_faillock uid=1000  exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
Dec 06 16:12:18 archie kernel: audit: type=1100 audit(1607289138.427:9081): pid=2225653 uid=1000 auid=1000 ses=861 msg='op=PAM:authentication grantors=? acct="myuser" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=failed'
Dec 06 16:12:18 archie kernel: audit: type=2206 audit(1607289138.427:9080): pid=2225653 uid=1000 auid=1000 ses=861 msg='pam_faillock uid=1000  exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
Dec 06 16:12:18 archie sudo[2225653]: pam_unix(sudo:auth): conversation failed
Dec 06 16:12:18 archie kernel: audit: type=1103 audit(1607289138.344:9079): pid=2225652 uid=0 auid=1000 ses=861 msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_env,pam_faillock acct="myuser" exe="/usr/bin/sshd" hostname=127>
Dec 06 16:12:18 archie audit[2225652]: CRED_ACQ pid=2225652 uid=0 auid=1000 ses=861 msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_env,pam_faillock acct="myuser" exe="/usr/bin/sshd" hostname=127.0.0.1 addr=127.0.0.1 termin>
Dec 06 16:12:18 archie kernel: audit: type=1105 audit(1607289138.341:9078): pid=2225649 uid=0 auid=1000 ses=861 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_unix,pam_permit,pam_mail,pam_systemd,pam_env acct="myuser>
Dec 06 16:12:18 archie audit[2225649]: USER_START pid=2225649 uid=0 auid=1000 ses=861 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_unix,pam_permit,pam_mail,pam_systemd,pam_env acct="myuser" exe="/usr/bin/sshd" host>
Dec 06 16:12:18 archie sshd[2225649]: pam_env(sshd:session): deprecated reading of user environment enabled
Dec 06 16:12:18 archie systemd[1]: Started Session 861 of user myuser.

Thanks for all the ideas guys. I feel more optimistic about tracking this down now.

Offline

#16 2020-12-07 17:47:29

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

I fixed the "sudo rsync" issue mentioned in post above, and I set up a different user account. However, the problem came back today on this other user account. So I am once again at a loss for what the issue might be.

Would it be worth removing/deleting, and reinstalling some packages?

Last edited by MountainX (2020-12-07 17:59:45)

Offline

#17 2021-07-15 23:53:51

ico
Member
Registered: 2011-01-25
Posts: 35

Re: sudo password not accepted until after I reboot

Apologies for bumping this thread.

But I have been facing the same issue. Password stops working if I try to sudo. Works fine after I reboot.

My system is also an SSH box. The only offbeat package I have installed is docker and I also run connect to a WireGuard network.

Offline

#18 2021-07-16 07:02:22

seth
Member
Registered: 2012-09-03
Posts: 49,981

Re: sudo password not accepted until after I reboot

Did you check about #7 and #8?

Offline

#19 2021-07-16 13:50:42

MountainX
Member
Registered: 2016-02-08
Posts: 371

Re: sudo password not accepted until after I reboot

ico wrote:

Apologies for bumping this thread.

But I have been facing the same issue. Password stops working if I try to sudo. Works fine after I reboot.

My system is also an SSH box. The only offbeat package I have installed is docker and I also run connect to a WireGuard network.

Look for any old scripts, cron jobs, etc. that could be triggering auth failures. That was the cause of my problem and it took me a while to find the old script I had forgotten about.

Offline

Board footer

Powered by FluxBB