[Solved] Bricked my arch linux build? (Kernel panic)

Sheltr · 2020-02-24 20:10:40

Alright, I've been using arch with KDE for a good solid 5 or 6 months now as my primary OS, but about two weeks ago or so, I think two of my sticks of ram died, maybe during an update or during at least before the update. Anyway, right after the update my computer pretty much just froze and crashed, I restarted only to be met up with a kernel panic, Tried putting in my drive with windows on it, kept having memory BSOD, I took out the two sticks of ram that I believe died, (the sticks are about 10 years old now, so they lasted pretty long for how much strain I put on them) managed to get windows running like that, but I still can't seem to get arch to boot at all, I am able to chroot into my partition via a USB with arch on it, But I am pretty much unable to do much of anything due to a nonstop "Parent Transid Verify Failed" error. Attempting to scrub it gets it caught up at about the same point that the error seems to be on, but most of the partition seems pretty much untouched, just that one probably key part seems to be damaged. I've already tried updating but trying anything in pacman seems to just give the same error along with a good chunk of other commands. Any suggestions or help to get me back up and running would be much appreciated.
I've spent quite a few hours on my own attempting different ways of trying to fix the problem but don't seem to be getting anywhere.

Also, I've just upgraded my hardware due to the failing parts from an old build, I have a new CPU motherboard and RAM, It did not change the problems in any way. Not sure if this info will be helpful but I'm just throwing it in here just in case.
I also believe that the update was a core arch-linux update but I don't remember at the time what else was updated. Thanks for reading.

Edit: File system is too far damaged due to the ram corruption, transferred some documents to my laptop and reinstalled arch. -end

Last edited by Sheltr (2020-03-09 21:33:02)

loqs · 2020-02-24 20:31:16

Have you run a S.M.A.R.T. self test on the system's storage devices?
Do you have backups you can restore from?
Edit:
The root filesystem is BTRFS and is corrupted preventing package re-installation?

Last edited by loqs (2020-02-24 20:35:48)

Sheltr · 2020-02-24 20:36:50

The SSD is pretty much brand new, and I do not have backups sadly (yes I know I should, and I was thinking about doing it probably a week before it happened but I ended up not doing it like an idiot.) but the health of the drive itself is perfectly fine.

Edit: The file system itself is fine I can see all the files are pretty much untouched besides the files it seems to be trying to call. Which seems to be integral to the system functioning, but I am able to chroot into it just fine, the problem is me trying to repair the damaged area. which seems to be less the 1% of the partition itself.

Last edited by Sheltr (2020-02-24 20:40:39)

loqs · 2020-02-24 20:43:53

Please post the dmesg from when you attempt to repair the damaged area and `smartctl -a /dev/X` where X is the system's storage device.

Sheltr wrote:

Edit: The file system itself is fine I can see all the files are pretty much untouched besides the files it seems to be trying to call. Which seems to be integral to the system functioning, but I am able to chroot into it just fine, the problem is me trying to repair the damaged area. which seems to be less the 1% of the partition itself.

How can the filesystem and storage device be fine when you can not access all the content on the filesystem?

Last edited by loqs (2020-02-24 20:47:20)

Sheltr · 2020-02-24 21:15:47

Sorry for the long wait, Booting in and out of windows takes a while and I don't have another computer to message you on.

It passed the smart test successfully.

And what I mean by 'is fine' I mean most of the files are not damaged or corrupted, it seems to be several key files that got damaged during an update. Most likely due to said ram issue I told you about, I'm guessing damaged memory mid update seemed to be the problem that caused this in the first place?

I'll be honest I'm not quite sure how I would go about posting the direct log from the smart test in my current situation?

loqs · 2020-02-24 21:18:12

Boot from the installation media, setup networking if needed, see the tip box from pastebin to post from the console.

Sheltr · 2020-02-24 21:33:08

Sorry postingthis from my phone. I'll just have to type the error out sorry.

BTRFS error (device sda4) parent transid verify failed on 152886591488 wanted 244491 found 244489

And and it's just this repeatedly. If I try to do much of anything.

Edit: also when I chroot in i get

Pidof: error while loading shared libraries: /usr/lib/librt.so.1: cannot read file data: input/output error.

Last edited by Sheltr (2020-02-24 21:42:30)

Sheltr · 2020-02-24 23:10:56

Kernel panic - not syncing: attempt to kill init! Exitcode=0x00007f00
CPU: 12 PID: 1 Comm: init not tainted 5.5.3-arch1-1 #1
Call Trace
Dump_stack+0x66/0x90
Panic+0x112/0x2e8
Do_exit.cold+0xa7/0xb0
? Task_work_run+0x3a/0xc1
Do_group_exit+0x7b/0xb0
__x64_sys_exit_group+0x14/0x20
Do_syscall_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f35a1de574e
Code: 89 fa b8 e7 00 00 00 be 3c 00 00 00 eb 14 0f 1f 44 00 00 89 d7 89 f0 0f 05 48 3d 00 00 ff ff 77 1a f4 89 d7 44 89 c0 0f 05 <48> 3d 00 f0 ff ff 76 e2 f7 d8 89 05 b2 f9 00 00 eb d8 f7 d8 89 05
RSP: 002b:00007ffec751a298 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0007f95a1d75a10 RCX: 00007f35a1de574e
RDX: 000000000000007f RSI: 000000000000003c RDI: 00000000000007f
RBP: 00007ffec751ac70 R08: 00000000000000e7 R09: 000000000000000
R10: 00007ffec751a1a8 R14: 0000000000000003 R15: 00007f35a1d75a40
Kernel offset: 0x13c00000 from 0xffffffff810000000 (relocation range: 0xffffffffff80000000-0xffffffffbffffffff)
End kernel panic - not syncing: attempt to kill init! Exit code=0x00007f00

d_fajardo · 2020-02-25 07:01:49

You said you have new CPU and Mobo. Are you using a GPU or reusing it? Could be memory from GPU that's the culprit.

Sheltr · 2020-02-25 07:13:34

I am reusing the GPU, which is an old GTX 970. I mean I suppose that is possible but I don't have a donor card right now to test that? I've noticed my system is still pretty unstable even after the upgrade which I was kind of surprised about, but I just kind of wrote that off as windows just being slow and crashing a lot like it always does, also I am able to run games without any real graphical errors, but they do seem to be chugging a bit more than I remember? Though this could just be all sort of placebo type effect. I'll have to do some tests to see if the video memory is still alright.

Edit: After some testing, I haven't had any real graphical errors or artifacting in games, so I don't believe that my GPU is the reason for the panic.

Last edited by Sheltr (2020-02-25 20:48:26)

Sheltr · 2020-03-01 03:14:45

I am still unable to update. When trying Pacman --root /mnt -Syyu I still get the same parent transid verify failed error along with a

BTRFS error (device sda4) in btrfs_finsh_order_io:3173: errno=-5 IO failure

I've done tests on the SSD which is only about 5 months old or so and it has no error.

I'm not sure where to go from here anymore. I can chroot into the system but I can't upgrade it.

loqs · 2020-03-01 11:19:25

Post the output of `smartctl -x /dev/sdX` where X is the system's storage device to a pastebin (see link in post #6).
After chrooting into the system post the journal from the system's last boot to the pastebin.
When the system upgrade fails please post the output of the failure and output from dmesg to the pastebin as well.

Last edited by loqs (2020-03-01 11:31:28)

Isaak.Aleksandrov · 2020-03-01 17:22:57

While I don't really have any good tips for how to recover a BTRFS filesystem, I'd still like to add that if you do not have backups, and there's anything of importance on the disk, then now's definitely the time to make a full clone the disk before doing anything else with it.

Something like Clonezilla would probably work just fine for the task. Of course, you will need to have another disk, or large enough USB/eHDD, to save the disk image to.

Sheltr · 2020-03-06 21:00:34

smartctl : ix.io/2ddg

I am unable to get the journal output sorry it seems that it needs the shared libraries that appear to be corrupted.

I can share other logs for pacman and such if that would help?

loqs · 2020-03-06 21:13:43

Instead of chrooting in and trying to get the journal setup the chroot but do not chroot in then:

journalctl --root=/mnt/ -b -1

Sheltr · 2020-03-07 21:55:55

ix.io/2dGq is the journalctl

I was having a lot of Wi-Fi problems at the time as my old Wi-Fi card was dying it has also been replaced if you are wondering about the nonstop disconnects.

Right when the computer updated pretty much everything just started crashing, and I was unable to do much of anything, I decided to just restart to see if that would help and upon restarting I was met with the kernel panic.

loqs · 2020-03-07 22:07:46

I would suggest asking on #btrfs IRC channel on freenode but that may not be viable without a working system.

Arch Linux

#1 2020-02-24 20:10:40

[Solved] Bricked my arch linux build? (Kernel panic)

#2 2020-02-24 20:31:16

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#3 2020-02-24 20:36:50

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#4 2020-02-24 20:43:53

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#5 2020-02-24 21:15:47

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#6 2020-02-24 21:18:12

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#7 2020-02-24 21:33:08

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#8 2020-02-24 23:10:56

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#9 2020-02-25 07:01:49

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#10 2020-02-25 07:13:34

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#11 2020-03-01 03:14:45

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#12 2020-03-01 11:19:25

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#13 2020-03-01 17:22:57

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#14 2020-03-06 21:00:34

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#15 2020-03-06 21:13:43

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#16 2020-03-07 21:55:55

Re: [Solved] Bricked my arch linux build? (Kernel panic)

#17 2020-03-07 22:07:46

Re: [Solved] Bricked my arch linux build? (Kernel panic)

Board footer