systemd seems to fail on boot

leadry · 2022-07-02 15:56:57

Systemd has just started failing after the battery on my laptop expired. When I boot now it first asks me for my FDE password and after that it gives me a bunch of FAILED systemd services.

I'm not sure how to debug this since I'm not able to scroll up I can't see the first error that happens. Here is a picture: https://ibb.co/pWY2qCV

Head_on_a_Stick · 2022-07-02 15:59:29

leadry wrote:

I'm not sure how to debug this since I'm not able to scroll up I can't see the first error that happens.

You can take a video and play it back frame-by-frame to see the full information.

But reading the journal contents from a live system seems like a better option: https://wiki.archlinux.org/title/System … al_to_view

EDIT: those permission errors suggest a read-only filesystem, possibly due to corruption.

Last edited by Head_on_a_Stick (2022-07-02 16:00:19)

leadry · 2022-07-02 21:42:26

I've tried to track down a USB for a live boot but no luck yet, nor is my camera good enough to catch the start. However how would I repair a read-only filesystem? fsck does not seem available from the grub commandline.

Last edited by leadry (2022-07-02 21:42:39)

seth · 2022-07-02 22:17:12

However how would I repair a read-only filesystem?

track down a USB for a live boot

leadry · 2022-07-02 23:14:17

I've filmed it and it seems the first FAILURE is "failed to mount tmp.mount"

Are you saying that I need a usb stick to repair a read-only filesystem?

seth · 2022-07-03 06:06:10

Yes.
There's also an fsck hook, but you don't get to add it from the rescue shell, facing a corrupted FS already.

Also

https://man.archlinux.org/man/core/e2fsprogs/fsck.ext4.8.en wrote:

Note that in general it is not safe to run e2fsck on mounted file systems. The only exception is if the -n option is specified, and -c, -l, or -L options are not specified. However, even if it is safe to do so, the results printed by e2fsck are not valid if the file system is mounted. If e2fsck asks whether or not you should check a file system which is mounted, the only correct answer is ``no''. Only experts who really know what they are doing should consider answering this question in any other way.

You'll have to fix that offline, ie. booting something else.

leadry · 2022-07-03 10:40:59

Thanks for the suggestion.

I've went and bought a new usb and can now boot into a live medium.

The path forward as I understand it is now something like.

1. Boot into live medium
2. Using cryptsetup to decrypt my FDE
3. Mounting the disk system from within the live boot (or should I avoid mounting it?)
4. Running fsck on the mounted system (or should I be using e2fsck here?)

Is this what you mean by fixing it offline?

Do you happen to know where there is a guide for something like this?

Last edited by leadry (2022-07-03 10:49:51)

loqs · 2022-07-03 11:04:41

From the live media I would suggest the first thing you do is run a S.M.A.R.T. self test on the drive. When that finishes post the output of smartctl -a $dev.
If the device is failing you would want to avoid mounting / repairing it until you had created an image of it assuming you wanted to recover data from the device before replacing it.

leadry · 2022-07-03 14:15:06

When I run
sudo smartctl -t short /dev/nvme0n1p3
it tells me NVMe device successfully opened

I then run
sudo smartctl -a /dev/nvme0n1p3
and get

SMART overall-health self-assessment test result: PASSED
Critical Warnings: 0x00
Available Spare: 100%
Available Spare Threshold: 10%
PercentageUsed: 0%
...
Unsafe Shutdowns: 44
...
Error Information Log Entries: 1
...
Error Information (NVMe Log 0x01, 15 of 256 entries)
No errors logged

I did not input my password to decrypt my FDE, should I have decrypted the drive with cryptsetup before running the smart check or would that endanger my data?

seth · 2022-07-03 14:36:24

smartctl -a /dev/nvme0

and that's hardly the *entire* output of the command.
You do not have to decrypt the device for this (but for an fsck later on you will - also to obtain, hopefully, the system journal of the installed system)

leadry · 2022-07-03 19:28:14

After having mounted my file system I can't find the error messages in the journal
I'm using journalctl to look at the mounted logs by

sudo journalctl -D /mnt/mounted_disk/var/log/journal/bdbf5b...b08b/

The last entries in the log pertain mostly to wifi shutting down and before that some of my other applications doing things. It does not contain any information about startup services failing.

Is there another place to look?

I've also tried writing to a file and it seems to work, so that seems to imply it's not a complete read-only filesystem.

seth · 2022-07-03 19:43:42

Journal doesn't sync to disk, probably fails.
Try to slow down the boot by passing "boot_delay=250" to the kernel.
Also make sure the "quiet" parameter isn't there.

But also post the complete smartctl -a output and run at lease a passive fsck on the partitions (no -a or -r)

leadry · 2022-07-03 20:09:24

Here is the total output of

smartctl -a /dev/nbmr0n1p3

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.5-300.fc36.x86_64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, [url=http://www.smartmontools.org]www.smartmontools.org[/url]

=== START OF INFORMATION SECTION ===
Model Number:                       WDC PC SN730 SDBQNTY-512G-1001
Serial Number:                      21154T801048
Firmware Version:                   11170101
PCI Vendor/Subsystem ID:            0x15b7
IEEE OUI Identifier:                0x001b44
Total NVM Capacity:                 512,110,190,592 [512 GB]
Unallocated NVM Capacity:           0
Controller ID:                      8215
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            001b44 8b4809cf75
Local Time is:                      Sun Jul  3 15:51:37 2022 EDT
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x1e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     84 Celsius
Critical Comp. Temp. Threshold:     88 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     5.50W       -        -    0  0  0  0        0       0
 1 +     3.50W       -        -    1  1  1  1        0       0
 2 +     3.00W       -        -    2  2  2  2        0       0
 3 -   0.0700W       -        -    3  3  3  3     4000   10000
 4 -   0.0035W       -        -    4  4  4  4     4000   40000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    1,541,897 [789 GB]
Data Units Written:                 2,546,686 [1.30 TB]
Host Read Commands:                 7,228,356
Host Write Commands:                57,178,831
Controller Busy Time:               37
Power Cycles:                       122
Power On Hours:                     29
Unsafe Shutdowns:                   44
Media and Data Integrity Errors:    0
Error Information Log Entries:      1
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

I am not sure how to run fsck correctly.
Running

sudo fsck /dev/nvme0n1p3

prints

fsck from util-linux 2.38-rc1

Running

sudo fsck /dev/nvme0n1

prints

/dev/nvme0n1 is in use.
e2fsck: Cannor continue, aborting

I've unmounted /mnt/my_disk
but I have not closed it with cryptsetup since I'm assuming it needs to be decrypted to be checked.

Last edited by leadry (2022-07-03 20:53:41)

seth · 2022-07-03 20:45:40

Please use code tags, https://bbs.archlinux.org/help.php#bbcode (edit your post accordingly)

Drive looks ok, you have to run fsck on the decrypted partition.

lsblk -f

leadry · 2022-07-03 20:49:17

After rebooting I'm getting an issue where it says

dracut-initqueue[903]: Configuration node devices/filter not found
dracut-initqueue[903]: Configuration node devices/global_filter not found
dracut-initqueue[903]: WARNING: File locking is disabled.
dracut-initqueue[903]: Volume group "name" not found
dracut-initqueue[903]: Cannot process volume group "name"

I'm thinking this might be due to me changing the name with

vgdisplay
vgrename <UUID> new_name

because I could not mount it as I got a "Unknown filesystem type "LVM2_member" error.

Last edited by leadry (2022-07-03 20:56:59)

seth · 2022-07-03 21:00:23

'm thinking this might be due to me changing the name

Yes, revert the name. [edit: english]

I could not mount it as I got a "Unknown filesystem type "LVM2_member" error.

Bogus mount command?
https://man.archlinux.org/man/core/util … mount.8.en

Last edited by seth (2022-07-03 21:00:53)

leadry · 2022-07-03 21:04:16

seth wrote:

Please use code tags, https://bbs.archlinux.org/help.php#bbcode (edit your post accordingly)
Drive looks ok, you have to run fsck on the decrypted partition.
lsblk -f

How should I do this?

First I decrypt the partition with

sudo cryptsetup luksOpen /dev/nvme0n1p3 realdisk

Then when I run

lsblk -f

NAME                  FSTYPE      FSVER            LABEL                 UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0                 squashfs    4.0                                                                                          
loop1                 ext4        1.0              Anaconda              626a3537-c047-4e1d-b6ad-a10855a3b5f9                  
├─live-rw             ext4        1.0              Anaconda              626a3537-c047-4e1d-b6ad-a10855a3b5f9      2.1G    71% /
└─live-base           ext4        1.0              Anaconda              626a3537-c047-4e1d-b6ad-a10855a3b5f9                  
loop2                                                                                                                          
└─live-rw             ext4        1.0              Anaconda              626a3537-c047-4e1d-b6ad-a10855a3b5f9      2.1G    71% /
sda                   iso9660     Joliet Extension Fedora-WS-Live-36-1-5 2022-05-04-21-36-58-00                                
├─sda1                iso9660     Joliet Extension Fedora-WS-Live-36-1-5 2022-05-04-21-36-58-00                       0   100% /run/initramfs/live
├─sda2                vfat        FAT16            ANACONDA              1F4B-F0F7                                             
└─sda3                hfsplus                      ANACONDA              28d52925-499c-3a1f-b20d-34e524d3b6ba                  
zram0                                                                                                                          [SWAP]
nvme0n1                                                                                                                        
├─nvme0n1p1           vfat        FAT32                                  AE8B-8A05                                             
├─nvme0n1p2           xfs                                                0fa196f3-b6b5-447b-99b8-49829b646320                  
└─nvme0n1p3           crypto_LUKS 2                                      5af04d49-0592-4272-a4f4-91fe9e7d40b2                  
  └─realdisk          LVM2_member LVM2 001                               200uDT-0t09-WajS-i2bQ-R7vU-GO12-pcWKFc                
    └─new_name-root xfs                                               f792e96e-19a2-41c3-816d-dc6c57d4f466

note that "new_name" is a name I gave with

vgrename <VG UUID> new_name

leadry · 2022-07-03 21:09:58

seth wrote:

'm thinking this might be due to me changing the name
Yes, revert the name. [edit: english]
I could not mount it as I got a "Unknown filesystem type "LVM2_member" error.
Bogus mount command?
https://man.archlinux.org/man/core/util … mount.8.en

Yes, this did it, reverting to an old name made the old errors come back.

The reason I did it in the first place is because it was suggested here: https://askubuntu.com/questions/766048/ … vm2-member

seth · 2022-07-03 21:30:08

"/dev/mapper/new_name-root" (or rather the now reverted one) is the device you want to fsck

leadry · 2022-07-03 22:24:59

seth wrote:

"/dev/mapper/new_name-root" (or rather the now reverted one) is the device you want to fsck

I had to run the following since it's an xfs filesystem.

xfs_repair /dev/mapper/name-root

Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
        - 22:18:53: zeroing log - 2560 of 2560 blocks done
        - scan filesystem freespace and inode maps...
        - 22:18:53: scanning filesystem freespace - 122 of 122 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - 22:18:53: scanning agi unlinked lists - 122 of 122 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 15
        - agno = 0
        - agno = 30
        - agno = 45
        - agno = 60
        - agno = 75
        - agno = 105
        - agno = 120
        - agno = 90
        - agno = 76
        - agno = 91
        - agno = 46
        - agno = 16
        - agno = 121
        - agno = 61
        - agno = 106
        - agno = 77
        - agno = 107
        - agno = 62
        - agno = 92
        - agno = 47
        - agno = 17
        - agno = 93
        - agno = 48
        - agno = 108
        - agno = 78
        - agno = 94
        - agno = 63
        - agno = 109
        - agno = 49
        - agno = 18
        - agno = 79
        - agno = 50
        - agno = 64
        - agno = 19
        - agno = 51
        - agno = 80
        - agno = 95
        - agno = 110
        - agno = 52
        - agno = 20
        - agno = 111
        - agno = 96
        - agno = 65
        - agno = 81
        - agno = 53
        - agno = 112
        - agno = 21
        - agno = 97
        - agno = 82
        - agno = 113
        - agno = 98
        - agno = 54
        - agno = 22
        - agno = 83
        - agno = 66
        - agno = 114
        - agno = 55
        - agno = 84
        - agno = 99
        - agno = 23
        - agno = 67
        - agno = 115
        - agno = 56
        - agno = 100
        - agno = 85
        - agno = 116
        - agno = 24
        - agno = 57
        - agno = 86
        - agno = 101
        - agno = 68
        - agno = 117
        - agno = 58
        - agno = 25
        - agno = 69
        - agno = 102
        - agno = 118
        - agno = 26
        - agno = 103
        - agno = 59
        - agno = 87
        - agno = 70
        - agno = 88
        - agno = 119
        - agno = 104
        - agno = 27
        - agno = 89
        - agno = 71
        - agno = 28
        - agno = 29
        - agno = 1
        - agno = 72
        - agno = 73
        - agno = 74
        - agno = 2
        - agno = 31
        - agno = 3
        - agno = 32
        - agno = 33
        - agno = 34
        - agno = 35
        - agno = 36
        - agno = 37
        - agno = 38
        - agno = 39
        - agno = 40
        - agno = 41
        - agno = 42
        - agno = 43
        - agno = 44
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - 22:18:54: process known inodes and inode discovery - 624512 of 621792 inodes done
        - process newly discovered inodes...
        - 22:18:54: process newly discovered inodes - 122 of 122 allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 22:18:54: setting up duplicate extent list - 122 of 122 allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 4
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - agno = 25
        - agno = 26
        - agno = 27
        - agno = 28
        - agno = 29
        - agno = 30
        - agno = 31
        - agno = 32
        - agno = 33
        - agno = 34
        - agno = 35
        - agno = 36
        - agno = 37
        - agno = 38
        - agno = 39
        - agno = 40
        - agno = 41
        - agno = 42
        - agno = 43
        - agno = 44
        - agno = 45
        - agno = 46
        - agno = 47
        - agno = 48
        - agno = 49
        - agno = 50
        - agno = 51
        - agno = 52
        - agno = 53
        - agno = 54
        - agno = 55
        - agno = 56
        - agno = 57
        - agno = 58
        - agno = 59
        - agno = 60
        - agno = 61
        - agno = 62
        - agno = 63
        - agno = 64
        - agno = 65
        - agno = 66
        - agno = 67
        - agno = 68
        - agno = 69
        - agno = 70
        - agno = 71
        - agno = 72
        - agno = 73
        - agno = 74
        - agno = 75
        - agno = 76
        - agno = 77
        - agno = 78
        - agno = 79
        - agno = 80
        - agno = 81
        - agno = 82
        - agno = 83
        - agno = 84
        - agno = 85
        - agno = 86
        - agno = 87
        - agno = 88
        - agno = 89
        - agno = 90
        - agno = 91
        - agno = 92
        - agno = 93
        - agno = 94
        - agno = 95
        - agno = 96
        - agno = 97
        - agno = 98
        - agno = 99
        - agno = 100
        - agno = 101
        - agno = 102
        - agno = 103
        - agno = 104
        - agno = 105
        - agno = 106
        - agno = 107
        - agno = 108
        - agno = 109
        - agno = 110
        - agno = 111
        - agno = 112
        - agno = 113
        - agno = 114
        - agno = 115
        - agno = 116
        - agno = 117
        - agno = 118
        - agno = 119
        - agno = 120
        - agno = 121
        - 22:18:54: check for inodes claiming duplicate blocks - 624512 of 621792 inodes done
Phase 5 - rebuild AG headers and trees...
        - 22:18:54: rebuild AG headers and trees - 122 of 122 allocation groups done
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
        - 22:18:54: verify and correct link counts - 122 of 122 allocation groups done
done

I'm not sure where to put the

boot_delay=250

flag. I tried putting it in the grub command line sequence as an argument to linuz however it ended up not booting.

seth · 2022-07-03 22:30:07

https://wiki.archlinux.org/title/Kernel_parameters#GRUB
nb. that this will slow down the boot process - make sure it's not just booting slowly.
You could try adding "boot_delay=1" to see whether it principally works.

leadry · 2022-07-03 22:42:43

It seems to principally work. I will get back to you with the output tomorrow, it's getting pretty late here.
Thank you for all of the help so far!

leadry · 2022-07-05 19:07:42

I ended up reinstalling the OS, thanks for all the help!

Arch Linux

#1 2022-07-02 15:56:57

systemd seems to fail on boot

#2 2022-07-02 15:59:29

Re: systemd seems to fail on boot

#3 2022-07-02 21:42:26

Re: systemd seems to fail on boot

#4 2022-07-02 22:17:12

Re: systemd seems to fail on boot

#5 2022-07-02 23:14:17

Re: systemd seems to fail on boot

#6 2022-07-03 06:06:10

Re: systemd seems to fail on boot

#7 2022-07-03 10:40:59

Re: systemd seems to fail on boot

#8 2022-07-03 11:04:41

Re: systemd seems to fail on boot

#9 2022-07-03 14:15:06

Re: systemd seems to fail on boot

#10 2022-07-03 14:36:24

Re: systemd seems to fail on boot

#11 2022-07-03 19:28:14

Re: systemd seems to fail on boot

#12 2022-07-03 19:43:42

Re: systemd seems to fail on boot

#13 2022-07-03 20:09:24

Re: systemd seems to fail on boot

#14 2022-07-03 20:45:40

Re: systemd seems to fail on boot

#15 2022-07-03 20:49:17

Re: systemd seems to fail on boot

#16 2022-07-03 21:00:23

Re: systemd seems to fail on boot

#17 2022-07-03 21:04:16

Re: systemd seems to fail on boot

#18 2022-07-03 21:09:58

Re: systemd seems to fail on boot

#19 2022-07-03 21:30:08

Re: systemd seems to fail on boot

#20 2022-07-03 22:24:59

Re: systemd seems to fail on boot

#21 2022-07-03 22:30:07

Re: systemd seems to fail on boot

#22 2022-07-03 22:42:43

Re: systemd seems to fail on boot

#23 2022-07-05 19:07:42

Re: systemd seems to fail on boot

Board footer