Unable to Boot into My Arch Linux Installation

Taughtimp2786 · 2022-10-15 06:52:51

(copy-pasted from reddit because I don't want to write all over again)

Some thing I would like to clear out:

I have been using this installation for more than a month. Reason being, around Three of my hard drives has gone corrupt because of using Arch Linux (at least I thinks so), and I can't diagnose them properly because my desktop computer can only hold one hard drive and I also have to remove the DVD Reader/Writer because with it being in my computer, my 16 GB ram wouldn't fit.

The installation not booting problem started just under this week.

Just before booting wouldn't work anymore, there were, I think you could call them 'Symptoms', which are:
1. qBittorrent would stop downloading a file and give an error that said `disk i/o error`
2. Some parts of Steam couldn't be clicked, for example, the button which when clicked should uninstall the chosen game but that button wouldn't work anymore
3. Calls on Signal couldn't be made because the `Start Call` button wouldn't work
4. Scroll would randomly stop working.

What I did try to do was, open `gnome-disks` as root just to see hard drive health, but I thought this error was user and permissions error (really stupid, I know) so I found an option called `Take ownership` after clicking on the 'hamburger menu' of the hard drive, and then I clicked on it (also stupid which I shouldn't have done). I am afraid that this is what is causing the problem.

Now the real problem comes when press on `Reboot`. This is what I get:

Starting version 251.5-1-arch
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Bluetooth service.
[FAILED] Failed to start D-Bus System Message Bus.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start D-Bus System Message Bus.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start Network Manager.
[DEPEND] Dependency failed for Network Manager Wait Online.
[FAILED] Failed to start D-Bus System Message Bus.
[FAILED] Failed to start User Login Management.

I booted into the live installation, and something related I found is that `systemd-timesyncd` (Network Time Synchronization) wouldn't start there either, so I had to manually start it. I mounted /dev/sda3 on /mnt and /dev/sda1 on /mnt/boot. After chrooting into /mnt, I did a system update.
Just a note: this was before I realized about the time sync service not starting error so I got many certificate invalid and keyring issues, which were solved after I synced time. Other thing is that I also got segmentation fault once during the update but that was also fixed by rebooting the installation medium. Once the system update had finished, I rebooted but I still couldn't boot into Arch Linux.

Some screenshots (4):
https://imgur.com/a/WaGMShQ

Last edited by Taughtimp2786 (2022-10-17 12:11:22)

seth · 2022-10-15 07:42:02

my hard drives has gone corrupt because of using Arch Linux (at least I thinks so)

You "thinks" wrong, but

1. qBittorrent would stop downloading a file and give an error that said `disk i/o error`

Whatever you're doing, this would be disk #4 - or, more likely, some bus issue ("cable")

This is what I get:

I'd tell you to get rid of the "quiet" kernel parameter, but …

Some screenshots

Theere FS errors and btrfs scrub isn't able to fix them.

=> https://wiki.archlinux.org/title/SMART
Boot some live distro like grml and check "smartctl -a" from there.
Also post a complete system journal (the last one you have) from the installed system, https://wiki.archlinux.org/title/System … al_to_view

Ftr, HDD is ST1000DM003-1ER162 - a seagate barracuda

Taughtimp2786 · 2022-10-15 10:35:58

Output of `smartctl -a`:
https://bin.wantguns.dev/p/Yu9yD4

journalctl:
https://termbin.com/bduwb

journalctl since 5 october (81.9):
https://termbin.com/t1w0w

Last edited by Taughtimp2786 (2022-10-15 10:40:53)

seth · 2022-10-15 13:02:13

 5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       16

Doesn't look too good.

The "journalctl" output is from Aug 26 and incomplete, please just obtain the last journal from that system.

Taughtimp2786 · 2022-10-15 14:14:29

Sorry for the late replies I am giving.

journalctl since 5 October:
https://termbin.com/vu63

journalctl since 5 October but reversed output:
https://termbin.com/oine

journalctl since 8 October:
https://termbin.com/nm83

journalctl since 10 October:
https://termbin.com/39nz

journalctl since 12 October:
https://termbin.com/jl8r

journalctl since 14 October:
https://termbin.com/qm1c

Sorry for this being too much but I also have to tell you that the issue started occurring on 9th October, so here are the journalctl(s) from that date. These are from 9 October to 10 October:
https://termbin.com/qs7zp
https://termbin.com/3yt3t (reversed output)

I also have to tell you that during the day when I started seeing the "symptoms", I opened dolphin and saw in the properties of the hard drive that the capacity of the hard drive was showing 128 TiB.

seth · 2022-10-15 14:45:38

The journals are corrupted or capped, they don't have any boot messages and mostly consist out of sshd warnings (failed logins from different IPs, all I checked were in China)

sudo journalctl -D /mnt/var/log/journal -b | curl -F 'file=@-' 0x0.st
sudo journalctl -D /mnt/var/log/journal -b-1 | curl -F 'file=@-' 0x0.st

Edit: in case this is a public facing ssh server, see https://wiki.archlinux.org/title/OpenSSH#Protection
Disable the root login, force public key authentication fail2ban/geoblocking can help to keep the traffic low - so can moving the server away from port #22 (it doesn't increase security, but "hides" it from annoying scriptkiddies)

Last edited by seth (2022-10-15 14:52:26)

Taughtimp2786 · 2022-10-15 15:38:40

https://0x0.st/ovnv.txt
https://0x0.st/ovnx.txt

About the SSH part, yes. I have known about this issue for long. It has been happening because someone got hold of my Dynamic DNS. I just removed the DNS and restarted my router so I got a new IP and hopefully I should stop getting these auth requests from the Chinese government asking me for files in my computer.
I will still work on securing my SSH and creating a fake SSH login on port 22 which only wastes the resources of the person trying to get in through that port (after I can finally boot into my computer).

seth · 2022-10-15 15:55:01

There're no device or bus errrors, but the FS is corrupted.
See https://wiki.archlinux.org/title/Btrfs# … n_recovery

Before that check

LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'

re-installing the corrupted packages might allow you to boot.

Taughtimp2786 · 2022-10-15 16:25:24

I first ran that pacman command you gave me. I managed to note down some packages that were listed until I got this: https://imgur.com/a/hPUvGb4
I got many BTRFS (mostly related to 'corrupt: <some number that increases its count each time the message appears>) and SQUASHFS. There were many outputs that said 'filesystem: <location to some file> (<in brackets, what error it was>)'. And then had a massive spam of SQUASHFS errors then kernel panic.

seth · 2022-10-15 16:31:40

You'll have to first repair the filesystem then.

Taughtimp2786 · 2022-10-15 17:54:56

I added btrfs to binaries in mkinitcpio.conf, then generated the image using `mkinitcpio -P`, then rebooted. Nothing has changed and the boot problem still persists.

seth · 2022-10-15 19:45:55

Adding that to the initramfs in and by itself does nothing, the point is to be able to use it w/o mounting the partition you need to fix.
You can however also just do that from the live system.

Taughtimp2786 · 2022-10-16 11:11:15

Didn't work.
To explain it clearly, I first did a `btrfs check /dev/sda3`, then I did a `btrfs check --repair /dev/sda3`, but didn't find any errors.
Then I mounted /dev/sda3 on /mnt and /dev/sda1 on /mnt/boot, then I arch-chrooted into /mnt then ran that pacman command you gave me, noted down all the names of packages that were outputted.
Then I reinstalled all the package with `pacman -S` except the AUR ones because I can't install AUR packages in root (says paru) and maybe because since they were AUR packages, they wouldn't need to be executed at boot.
Here is the list (the ones with dot in front of them are the AUR packages):

bluez
bluez-utils
ddclient
dnscrypt-proxy
filesystem
SQUASHFS
fwupd
ghc-libs
go
glibc
grub
intel-ucode
java-runtime-common
jdownloader2 .
jre-openjdk-headless
khtml
lib32-gcc-libs
libutempter
libva-mesa-driver
cuda
linux-headers
lsp-plugins
nerd-fonts-fira-code .
mkinitcpio
nvidia-utils
pacman-mirrorlist
plasma-desktop
plasma-nm
plasma-workspace
plasma-workspace-wallpapers
python-gobject
python-google-api-core
python-google-api-python-client
python-scipy
rabbitmq
shadow
signal-desktop
systemd
sudo
tusk .
vlc
texlive-core
wps-office-mui-zh-cn .
xorg-xinit

Last edited by Taughtimp2786 (2022-10-16 12:13:30)

seth · 2022-10-16 13:00:44

You're probably still getting btrfs errors, though?

Taughtimp2786 · 2022-10-17 04:29:04

Ok so I ran,`btrfs check --repair /dev/sda3` again (it has been running since 05:00 PM yesterday and today while I am writing this reply, it is 09:55 AM).
The only thing being outputted in the console is this:

Deleting bad dir index [5343,96,3189] root 5

The three numbers itself are random and change from time to time, but the third number changes the fastest (currently), I think the second number has been 96 since last night, and the first number only changes periodically.

Edit (10:00 PM)
The first number changed to 8703 for a while and then changed to 5376, 14176, 14179, 14182, and then back to 5376 and then changed again.
The second number is still 96.
When the first number changes, the third number too changes to something and then keeps on increasing till the first number changes, and then starts from new third number again.

Last edited by Taughtimp2786 (2022-10-17 04:33:32)

Taughtimp2786 · 2022-10-17 06:26:52

I got a segmentation fault. I feel sad now.
Now when I boot, I get this:

:: running early hook [udev]
Starting version 251.6-1-arch
:: running hook [udev]
:: Triggering uevents...
:: performing fsck on '/dev/sda3'
:: mounting '/dev/sda3' on real root
:: running cleanup hook [udeu]

Welcome to Arch Linux!

[!!!!!!] Failed to load rescue.target.
[ 7.165104] systemd[1]: Freezing execution.

I feel even more sad now.

Last edited by Taughtimp2786 (2022-10-17 06:27:15)

seth · 2022-10-17 06:46:34

If the btrfs errors are gone, you'll probably have to re-install all packages to replace the desintegrated files (offline, ie. from a live system boot)
https://wiki.archlinux.org/title/Pacman … dependency?

Then see what of your data is left …

Taughtimp2786 · 2022-10-17 11:04:35

pacman won't proceed to installation and only outputs errors like:

<package name>: <location to some file related to the package> exists in filesystem

Last edited by Taughtimp2786 (2022-10-17 11:05:05)

V1del · 2022-10-17 11:56:50

use --overwrite * as an additional flag.to the -S command

Taughtimp2786 · 2022-10-17 12:07:12

Example command please
I tried the following:

pacman -S --overwrite * $(pacman -Qnq)
pacman --overwrite * -S $(pacman -Qnq)
pacman -S $(pacman -Qnq) --overwrite *

None worked for me

Last edited by Taughtimp2786 (2022-10-17 12:07:38)

seth · 2022-10-17 12:40:26

You need to escape or single-quote the asterisk.

Taughtimp2786 · 2022-10-17 12:47:02

I figured out the command. It was:

pacman -S --overwrite \* $(pacman -Qnq)

But now the output of that command is:

error: no targets specified (use -h for help)

and so when I use `pacman -Q` or `pacman -Qnq`, the output returns empty.

seth · 2022-10-17 13:00:49

Means that the output of "pacman -Qnq" is empty.
Are you doing this from with a chroot?
Because otherwise you forgot sth. like "--sysroot /mnt"

Taughtimp2786 · 2022-10-17 13:10:42

Yes. I did every command in chroot except the BTRFS command.
and something I forgot to tell you was, the pacman command *did* work before, except it failed for 5 times as I kept retrying, and at the last attempt, I got BTRFS error. So I ran `btrfs check --repair /dev/sda3` **AFTER I EXITED** from chroot, but this time it was much much faster that it finished in a few minutes. I rebooted into live image again, mounted, chrooted, and that's when I ran the pacman command that it didn't work.

seth · 2022-10-17 13:21:08

Doing this from within the chroot isn't a good idea since you cannot trust the integrity of the chrooted system.
Also you should check smartctl again to see whether the Reallocated_Sector_Ct went up.

You also might have lost the entire pacman database in which case you'd have to https://wiki.archlinux.org/title/Pacman … l_database
It might be easier to recover as much private data as you can, ensure the drive itslf is healthy and setup a new system.

You also want to reflect on your usage patterns and what may cause the FS issues (notably if you shutdown the system by pulling the AC or so…)

Arch Linux

#1 2022-10-15 06:52:51

Unable to Boot into My Arch Linux Installation

#2 2022-10-15 07:42:02

Re: Unable to Boot into My Arch Linux Installation

#3 2022-10-15 10:35:58

Re: Unable to Boot into My Arch Linux Installation

#4 2022-10-15 13:02:13

Re: Unable to Boot into My Arch Linux Installation

#5 2022-10-15 14:14:29

Re: Unable to Boot into My Arch Linux Installation

#6 2022-10-15 14:45:38

Re: Unable to Boot into My Arch Linux Installation

#7 2022-10-15 15:38:40

Re: Unable to Boot into My Arch Linux Installation

#8 2022-10-15 15:55:01

Re: Unable to Boot into My Arch Linux Installation

#9 2022-10-15 16:25:24

Re: Unable to Boot into My Arch Linux Installation

#10 2022-10-15 16:31:40

Re: Unable to Boot into My Arch Linux Installation

#11 2022-10-15 17:54:56

Re: Unable to Boot into My Arch Linux Installation

#12 2022-10-15 19:45:55

Re: Unable to Boot into My Arch Linux Installation

#13 2022-10-16 11:11:15

Re: Unable to Boot into My Arch Linux Installation

#14 2022-10-16 13:00:44

Re: Unable to Boot into My Arch Linux Installation

#15 2022-10-17 04:29:04

Re: Unable to Boot into My Arch Linux Installation

#16 2022-10-17 06:26:52

Re: Unable to Boot into My Arch Linux Installation

#17 2022-10-17 06:46:34

Re: Unable to Boot into My Arch Linux Installation

#18 2022-10-17 11:04:35

Re: Unable to Boot into My Arch Linux Installation

#19 2022-10-17 11:56:50

Re: Unable to Boot into My Arch Linux Installation

#20 2022-10-17 12:07:12

Re: Unable to Boot into My Arch Linux Installation

#21 2022-10-17 12:40:26

Re: Unable to Boot into My Arch Linux Installation

#22 2022-10-17 12:47:02

Re: Unable to Boot into My Arch Linux Installation

#23 2022-10-17 13:00:49

Re: Unable to Boot into My Arch Linux Installation

#24 2022-10-17 13:10:42

Re: Unable to Boot into My Arch Linux Installation

#25 2022-10-17 13:21:08

Re: Unable to Boot into My Arch Linux Installation

Board footer