You are not logged in.

#1 2021-06-24 19:27:19

stefan
Member
Registered: 2013-03-22
Posts: 81

[solved] Upgrade with unnoticed unmounted /boot broke my system

Hello

Today, upgrading with `pacman -Syu` broke my system.  The reboot ended in an emergency mode, and allowed me to log in as root.

I do not know whether this is a bug, or whether I've messed something up.

The upgrade was (/var/log/pacman.log):

[2021-06-24T20:17:52+0200] [ALPM] transaction started
[2021-06-24T20:17:52+0200] [ALPM] upgraded libldap (2.4.58-3 -> 2.4.59-1)
[2021-06-24T20:17:53+0200] [ALPM] upgraded bind (9.16.17-1 -> 9.16.18-1)
[2021-06-24T20:17:53+0200] [ALPM] upgraded exiv2 (0.27.3-1 -> 0.27.4-2)
[2021-06-24T20:17:53+0200] [ALPM] upgraded mesa (21.1.2-1 -> 21.1.3-1)
[2021-06-24T20:17:53+0200] [ALPM] upgraded libva (2.11.0-1 -> 2.12.0-1)
[2021-06-24T20:17:53+0200] [ALPM] upgraded libcanberra (0.30+2+gc0620e4-4 -> 0.30+2+gc0620e4-5)
[2021-06-24T20:17:53+0200] [ALPM] upgraded sqlite (3.35.5-2 -> 3.36.0-1)
[2021-06-24T20:17:54+0200] [ALPM] upgraded firefox (89.0.1-1 -> 89.0.2-1)
[2021-06-24T20:17:54+0200] [ALPM] upgraded gnupg (2.2.28-1 -> 2.2.28-2)
[2021-06-24T20:17:54+0200] [ALPM] upgraded graphviz (2.47.2-1 -> 2.47.3-1)
[2021-06-24T20:17:55+0200] [ALPM] upgraded linux-lts (5.10.44-1 -> 5.10.45-1)
[2021-06-24T20:17:58+0200] [ALPM] upgraded linux-lts-headers (5.10.44-1 -> 5.10.45-1)
[2021-06-24T20:17:58+0200] [ALPM] upgraded nvidia-lts (1:465.31-8 -> 1:465.31-9)
[2021-06-24T20:17:58+0200] [ALPM] upgraded python-pip (20.3.2-1 -> 20.3.4-1)
[2021-06-24T20:17:59+0200] [ALPM] upgraded qt5-base (5.15.2+kde+r196-1 -> 5.15.2+kde+r200-1)
[2021-06-24T20:17:59+0200] [ALPM] upgraded qt5-declarative (5.15.2+kde+r28-1 -> 5.15.2+kde+r29-1)
[2021-06-24T20:17:59+0200] [ALPM] upgraded qt5-script (5.15.4-1 -> 5.15.5-2)
[2021-06-24T20:17:59+0200] [ALPM] upgraded tor (0.4.5.9-1 -> 0.4.6.5-1)
[2021-06-24T20:17:59+0200] [ALPM] transaction completed

The journal reported

Jun 24 20:19:53 tauhou systemd[1]: boot.mount: Directory /boot to mount over is not empty, mounting anyway.
Jun 24 20:19:53 tauhou systemd[1]: Mounting /boot...
Jun 24 20:19:53 tauhou mount[536]: mount: /boot: unknown filesystem type 'vfat'.
Jun 24 20:19:53 tauhou systemd[1]: Failed to mount /boot.

and `modprobe` would not find a `vfat` module.  Also, was surprised to find `/boot` not empty.

I decided to manually downgrade

pacman -U linux-lts{,-headers}-5.10.44-1-x86_64.pkg.tar.zst

which I'm suprised that it worked, given thet `/boot` was not mounted, and not mountable due to missing vfat module.

I repeated the previous step after rebooting, and making sure that `/boot` is mounted.  Then I also deleted the contents of an unmounted `/boot` which also contained initramfs and vmlinuz files (I did not check where from).

I also had to

pacman -U 'nvidia-lts-1:465.31-8-x86_64.pkg.tar.zst'

to get X working again.

I've assumed `vfat` to be a requirement of that UEFI stuff.  Is it going away?

Kind regards,
Stefan

Last edited by stefan (2021-06-24 21:34:56)

Offline

#2 2021-06-24 19:50:43

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 29,502
Website

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

stefan wrote:

Then I also deleted the contents of an unmounted `/boot` which also contained initramfs and vmlinuz files (I did not check where from).

/boot wasn't mounted for the upgrade. That's why you had those stray files, and your modules were not available.


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#3 2021-06-24 21:24:32

stefan
Member
Registered: 2013-03-22
Posts: 81

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

Hi Jason,

jasonwryan wrote:

/boot wasn't mounted for the upgrade. That's why you had those stray files, and your modules were not available.

Ah ja, makes sense.  Kinda obvious, now that you've mentioned it.  ;-)

So I wonder why `/boot` was not mounted, and how I could protect against that happening again.

Looking backwards in the journal, unmounting `/boot` always comes just before shutdown, without error messages.  Except for this one:

Jun 24 08:24:24 tauhou systemd[1]: Unmounting /boot...
Jun 24 08:24:24 tauhou kernel: FAT-fs (sdb1): unable to read boot sector to mark fs as dirty
Jun 24 08:24:24 tauhou systemd[1]: boot.mount: Deactivated successfully.
Jun 24 08:24:24 tauhou systemd[1]: Unmounted /boot.
Jun 24 08:24:28 tauhou systemd-logind[578]: Power key pressed.
Jun 24 08:24:28 tauhou systemd-logind[578]: Powering Off...
Jun 24 08:24:28 tauhou systemd-logind[578]: System is powering down.

That was this morning, I pressed PWR button right after boot (when login prompt was visible).  An unusual thing for me to do.

The next boot reported “not properly unmounted”

Jun 24 10:39:45 tauhou systemd[1]: Mounting /boot...
Jun 24 10:39:45 tauhou systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Jun 24 10:39:45 tauhou kernel: asus_wmi: ASUS WMI generic driver loaded
Jun 24 10:39:45 tauhou kernel: FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Jun 24 10:39:45 tauhou kernel: asus_wmi: Initialization: 0x0
Jun 24 10:39:45 tauhou kernel: asus_wmi: BIOS WMI version: 0.9
Jun 24 10:39:45 tauhou kernel: asus_wmi: SFUN value: 0x0
Jun 24 10:39:45 tauhou kernel: eeepc-wmi eeepc-wmi: Detected ASUSWMI, use DCTS
Jun 24 10:39:45 tauhou kernel: input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input3
Jun 24 10:39:45 tauhou systemd[1]: Mounted /boot.

but it still says “Mounted /boot”.  The following umount did not appear at shutdown, but I do not know what could have possibly triggered the umount:

Jun 24 11:16:55 tauhou systemd[1]: Unmounting /boot...
Jun 24 11:16:55 tauhou systemd[724]: boot.mount: Deactivated successfully.
Jun 24 11:16:56 tauhou systemd[1]: boot.mount: Deactivated successfully.
Jun 24 11:16:56 tauhou systemd[1]: Unmounted /boot.

The system kept running, amongst others

Jun 24 20:19:37 tauhou mkinitcpio[19064]: ==> Build complete.

indicating that, indeed, there was an update on umounted `/boot`.

I probably won't find out what has triggered unmounting of `/boot`. But how could I prevent an update running in that case?  Is there a canonical place where I could place a `mountpoint /boot/` test, before such an update is executed?  Would a mkinitcpio(8) hook be appropriate?

Thanks
Stefan

Last edited by stefan (2021-06-24 21:26:38)

Offline

#4 2021-06-24 21:31:56

seth
Member
Registered: 2012-09-03
Posts: 22,358

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

Jun 24 08:24:24 tauhou kernel: FAT-fs (sdb1): unable to read boot sector to mark fs as dirty
…
Jun 24 10:39:45 tauhou kernel: FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

Is /boot in your fstab?

Offline

#5 2021-06-24 21:37:16

stefan
Member
Registered: 2013-03-22
Posts: 81

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

seth wrote:

Is /boot in your fstab?

Yes:

/dev/sdb1       /boot           vfat            user,rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro      0 2

can't remember typing that, was probably generated.

Offline

#6 2021-06-24 21:46:48

seth
Member
Registered: 2012-09-03
Posts: 22,358

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

Did you check the SMART data for the disk?
What's the context around the umount journal entry?

Offline

#7 2021-06-24 21:55:58

stefan
Member
Registered: 2013-03-22
Posts: 81

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

seth wrote:

Did you check the SMART data for the disk?

Woha, you scare mae.  It's a new SSD (couple of weeks old), I have not checked.

seth wrote:

What's the context around the umount journal entry?

I've chosen the context based on the time gaps after the first, and before the last line.  Otherwise, these journals are vast...

Jun 24 10:54:50 tauhou audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 24 11:15:50 tauhou sshd[918]: Received disconnect from 192.168.0.161 port 34094:11: disconnected by user
Jun 24 11:15:50 tauhou kernel: audit: type=1106 audit(1624526150.128:64): pid=916 uid=0 auid=1000 ses=3 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_unix,pam_permit,pam_mail,pam_systemd,pam_env acct="sk" exe="/usr/bin/sshd" hostname=192.168.0.161 addr=192.168.0.161 terminal=ssh res=success'
Jun 24 11:15:50 tauhou kernel: audit: type=1104 audit(1624526150.128:65): pid=916 uid=0 auid=1000 ses=3 msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_env,pam_faillock acct="sk" exe="/usr/bin/sshd" hostname=192.168.0.161 addr=192.168.0.161 terminal=ssh res=success'
Jun 24 11:15:50 tauhou audit[916]: USER_END pid=916 uid=0 auid=1000 ses=3 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_unix,pam_permit,pam_mail,pam_systemd,pam_env acct="sk" exe="/usr/bin/sshd" hostname=192.168.0.161 addr=192.168.0.161 terminal=ssh res=success'
Jun 24 11:15:50 tauhou audit[916]: CRED_DISP pid=916 uid=0 auid=1000 ses=3 msg='op=PAM:setcred grantors=pam_shells,pam_faillock,pam_permit,pam_env,pam_faillock acct="sk" exe="/usr/bin/sshd" hostname=192.168.0.161 addr=192.168.0.161 terminal=ssh res=success'
Jun 24 11:15:50 tauhou systemd[1]: session-3.scope: Deactivated successfully.
Jun 24 11:15:50 tauhou sshd[918]: Disconnected from user sk 192.168.0.161 port 34094
Jun 24 11:15:50 tauhou systemd[1]: session-3.scope: Consumed 13.456s CPU time.
Jun 24 11:15:50 tauhou sshd[916]: pam_unix(sshd:session): session closed for user sk
Jun 24 11:15:50 tauhou systemd-logind[494]: Session 3 logged out. Waiting for processes to exit.
Jun 24 11:15:50 tauhou systemd-logind[494]: Removed session 3.
Jun 24 11:16:55 tauhou kernel: usb 8-2: USB disconnect, device number 2
Jun 24 11:16:55 tauhou systemd[1]: Stopped target Local File Systems.
Jun 24 11:16:55 tauhou systemd[1]: Unmounting /boot...
Jun 24 11:16:55 tauhou systemd[724]: boot.mount: Deactivated successfully.
Jun 24 11:16:56 tauhou systemd[1]: boot.mount: Deactivated successfully.
Jun 24 11:16:56 tauhou systemd[1]: Unmounted /boot.
Jun 24 13:46:58 tauhou dbus-daemon[490]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.15' (uid=0 pid=1022 comm="sshd: sk [priv]     ")

Offline

#8 2021-06-24 22:04:38

seth
Member
Registered: 2012-09-03
Posts: 22,358

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

You can post it to ix.io, see 1st link below

Jun 24 11:16:55 tauhou kernel: usb 8-2: USB disconnect, device number 2
Jun 24 11:16:55 tauhou systemd[1]: Stopped target Local File Systems.

Some usb device got removed, the local filesystem target stops and /boot gets unmounted
=> what was that device?

Offline

#9 2021-06-24 22:04:50

stefan
Member
Registered: 2013-03-22
Posts: 81

Re: [solved] Upgrade with unnoticed unmounted /boot broke my system

Oh, I see.

It should have never beed /dev/sdb1 that's mounted to /boot.  It should have been /dev/sda1.  It just happens to be the case that another USB device is typically present during boot, but it was never intended to actually contain /boot.  And I occasionally do unplug it...

Thanks for pointing me in the right direction seth, although I did not notice right away:  The entry in /etc/fstab is wrong, must refer to /dev/sda1 instead.

Thank you all!


This is one of the F***ups where I'm really angry at myself.  Sorry for wasting your time...

Last edited by stefan (2021-06-24 22:07:09)

Offline

Board footer

Powered by FluxBB