[SOLVED] No systemctl commands work

oonebaddog · 2024-07-28 00:25:20

Hello,

I was installing

flatpak

when my computer crashed earlier today. I thought nothing of it at the time, hardware rebooted and when I got back almost everything was broken.

I tried connecting to the internet. Met with `Connection Refused`. Tried running Hyprland:

$XDG_RUNTIME_DIR is unset

Did some light Googling, tried running a few systemctl commands like `systemctl status NetworkManager.service` and still got `Connection refused` or something similar. Sorry I can't provide the actual logs/commands currently as I'm away (and typing this on my phone), but I couldn't find any other sources as to why this might be happening. Any help would be appreciated. Thanks

Last edited by oonebaddog (2024-07-30 01:03:31)

seth · 2024-07-28 07:50:47

Boot the install iso, mount the installation root into /mnt and run

LC_ALL=C pacman --root=/mnt -Qkk | grep -v ', 0 altered files' | grep -v backup

Everything there showing up w/ a broken or missing mtree needs to be re-installed "--dbonly" and then again properly along everything else w/ suspicious file deviations (every deviation in /usr/{lib*,bin} is bad, some changes in /etc config files are perfectly normal the rest needs casewise judgement)
To re-install from the iso run

LC_ALL=C pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Sy <list of packages>

if the system looks fubar, you can also re-install all packages
but you'll still HAVE TO fix the mtree's first by installing at least those pacakges --dbonly, otherwise you'll run into file conflicts.

Don't forget to mount the boot partition into /mnt/boot if you have one.

oonebaddog · 2024-07-28 18:43:08

So I tried running

LC_ALL=C pacman --root=/mnt -Qkk | grep -v ', 0 altered files' | grep -v backup

And got

failed to initialize alpm library
(root: /mnt, dbpath: /mnt/var/lib/pacman)
Could not find or read directory

Did some more googling, it seems the main cause of this issue is /var/lib/pacman doesn't exist/have exec permissions, but I'm not sure that's the case for me

# ls -l /var/lib/pacman

drwxr-xr-x 986 root root 45056 Jul 27 16:09 local
drwxr-xr-x   2 root root  4096 Jul 28 08:14 sync

I figured I might be in a fubar situation, I tried running

pacman -Qnq | pacman --dbonly -S -

Met with a wall of

Failed retrieving file '<file name>' from <mirror>: Could not resolve host <mirror>

seth · 2024-07-28 19:25:02

And got

This is from the iso and you mounted the root of the installed system into /mnt??

lsblk -f
ls /mnt

Met with a wall of

You didn't configure a network connection.

oonebaddog · 2024-07-28 22:26:34

This is from the iso and you mounted the root of the installed system into /mnt??

Yeaaa, so the problem was you said that and my brain went "do the mount and also arch-chroot". *face meet palm*

So I fixed that. The other command works fine but theres a couple hundred lines of output. How do I determine which files have broken mtrees?

oonebaddog · 2024-07-28 22:30:15

How do I determine which files have broken mtrees?

3 seconds later...

LC_ALL=C pacman --root=/mnt -Qkk | grep -v ', 0 altered files' | grep -v backup > out

cat out | grep mtree

oonebaddog · 2024-07-29 00:49:32

I ran

LC_ALL=C pacman --root=/mnt -Qkk | grep -v ', 0 altered files' | grep -v backup > out

cat out | grep mtree

There was a list of about 7 packages. I then did

LC_ALL=C pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg --dbonly -Sy <list of packages>
LC_ALL=C pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Sy <list of packages>

I figured I fixed the issue, so I rebooted normally, but same problems

seth · 2024-07-29 06:07:36

There might be packges broken that still had perfectly fine mtrees:

LC_ALL=C pacman --root=/mnt -Qkk | grep -v ', 0 altered files' | grep -v backup > out
cat out | curl -F 'file=@-' 0x0.st

pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Sy <those packages>
pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Syu # to clear the stage from partial updates caused by the delay between the various steps

WorMzy · 2024-07-29 07:45:00

pacman ... --dbonly -Sy <list of packages>

Note that this can result in old files being 'left behind', as they are no longer part of the updated package, but they weren't cleaned up by the upgrade process. Where possible, it's better to recreate the same package version's db entry as was corrupted (using paclog and/filesystem inspection to determine what that version is). If those specific packages are no longer in your local package cache, the Arch Linux Archive will likely have them.

This may help in this situation where the damage has already been done.

oonebaddog · 2024-07-29 18:34:41

Can you explain what this does?

cat out | curl -F 'file=@-' 0x0.st

I understand I'm sending the file to that address. But why? What should I be expecting to be returned?

loqs · 2024-07-29 18:53:36

oonebaddog wrote:

Can you explain what this does?
cat out | curl -F 'file=@-' 0x0.st
I understand I'm sending the file to that address. But why? What should I be expecting to be returned?

A URL will be returned. The pastebin will be hosting the contents of the file out at that URL.

LC_ALL=C pacman --root=/mnt -Qkk | grep -v ', 0 altered files' | grep -v backup > out

Will return all installed packages that have altered files. The altered files could be due to file-system corruption.

oonebaddog · 2024-07-29 19:54:45

Ah ok thanks.

cat out | curl -F 'file=@-' 0x0.st
http://0x0.st/Xf13.csv

I checked that url out of curiosity, it looks like some error.

pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Sy $(cat out | cut -d':' -f1)
<package name> is up to date -- reinstalling
...

pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Syu $(cat out | cut -d':' -f1)
warning: <package name> is up to date -- reinstalling
...

Last edited by oonebaddog (2024-07-29 19:55:10)

seth · 2024-07-29 20:20:29

I checked that url out of curiosity, it looks like some error.

Because it's http://0x0.st/XfI3.csv - you need a better terminal font

It's 774 packages, just re-install all packages

"is up to date -- reinstalling" is ok, afterwards that list should be *drastically* shorter?

oonebaddog · 2024-07-29 20:59:17

you need a better terminal font

Sheesh, maybe I just need glasses, even with what you posted I can't tell if that's an uppercase I or a lower case L. Also... you can do that from the iso?! That would actually help so much since the default font for me is eye-strainingly small.

Reinstalled all packages with

arch-chroot /mnt
pacman -Qnq | pacman -S -

Exited after and ran

LC_ALL=C pacman --root=/mnt -Qkk | grep -v ', 0 altered files' | grep -v backup > out

afterwards that list should be *drastically* shorter?

Edit:
We're at 773 now lol. I think the other files are
themes, icons, zsh stuff though AUR packages. Still looking...

Last edited by oonebaddog (2024-07-29 21:03:34)

seth · 2024-07-29 21:15:17

This alone is lethal:

systemd: 1543 total files, 1229 altered files
systemd-libs: 817 total files, 805 altered files
systemd-sysvcompat: 14 total files, 4 altered files
glibc: 1614 total files, 1481 altered files
ncurses: 3935 total files, 3059 altered files

I even doubt the chroot worked but it's unlikely that pacman did?

pacman --root=/mnt -Qnq |  pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -S
pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Syu

pay close attention to any errors.

oonebaddog · 2024-07-29 21:35:21

Oof. Tried

pacman --root=/mnt -Qnq |  pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -S -

error: zstd: signature from <name and email> is unknown trust
File <file> is corrupted invalid or corrupted PGP signature

Found this arch forum post from a few years ago. Just not sure if that would help here.

seth · 2024-07-29 21:57:39

Is the archiso a bit dated?

oonebaddog · 2024-07-29 21:59:01

Definitely over a year old

seth · 2024-07-29 22:02:10

You can try to "pacman -Sy archlinux-keyring" on the iso first or "pacman-key --refresh-keys" (no chroot, nor "--root" for either of this) but the reliable, fast fix would be to use a recent install iso.

oonebaddog · 2024-07-29 22:49:52

Updated keyring.

pacman --root=/mnt -Qnq |  pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -S -
pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -Syu

Packages updated. Cool. Reboot:

Failed to mount /boot

seth · 2024-07-29 22:53:11

Forgot to mount the /boot partition into /mnt/boot before the update?
Do so and re-install the kernel.

oonebaddog · 2024-07-29 23:44:53

Wooo! I can at least login to my WM. New problem though

# systemctl list-jobs
No jobs running

No internet. Tried

# ip link set wlan0 up
RTNETLINK answers: operation not permitted

oonebaddog · 2024-07-30 01:01:45

Fixed. Forgot to disable iwd.service. marking as resolved.

Thanks for the help!

Arch Linux

#1 2024-07-28 00:25:20

[SOLVED] No systemctl commands work

#2 2024-07-28 07:50:47

Re: [SOLVED] No systemctl commands work

#3 2024-07-28 18:43:08

Re: [SOLVED] No systemctl commands work

#4 2024-07-28 19:25:02

Re: [SOLVED] No systemctl commands work

#5 2024-07-28 22:26:34

Re: [SOLVED] No systemctl commands work

#6 2024-07-28 22:30:15

Re: [SOLVED] No systemctl commands work

#7 2024-07-29 00:49:32

Re: [SOLVED] No systemctl commands work

#8 2024-07-29 06:07:36

Re: [SOLVED] No systemctl commands work

#9 2024-07-29 07:45:00

Re: [SOLVED] No systemctl commands work

#10 2024-07-29 18:34:41

Re: [SOLVED] No systemctl commands work

#11 2024-07-29 18:53:36

Re: [SOLVED] No systemctl commands work

#12 2024-07-29 19:54:45

Re: [SOLVED] No systemctl commands work

#13 2024-07-29 20:20:29

Re: [SOLVED] No systemctl commands work

#14 2024-07-29 20:59:17

Re: [SOLVED] No systemctl commands work

#15 2024-07-29 21:15:17

Re: [SOLVED] No systemctl commands work

#16 2024-07-29 21:35:21

Re: [SOLVED] No systemctl commands work

#17 2024-07-29 21:57:39

Re: [SOLVED] No systemctl commands work

#18 2024-07-29 21:59:01

Re: [SOLVED] No systemctl commands work

#19 2024-07-29 22:02:10

Re: [SOLVED] No systemctl commands work

#20 2024-07-29 22:49:52

Re: [SOLVED] No systemctl commands work

#21 2024-07-29 22:53:11

Re: [SOLVED] No systemctl commands work

#22 2024-07-29 23:44:53

Re: [SOLVED] No systemctl commands work

#23 2024-07-30 01:01:45

Re: [SOLVED] No systemctl commands work

Board footer