Arch error on start up

mrconfused · 2024-01-31 07:51:45

Hello, after my last pacman upgrade I have started getting a very detailed error on startup:

(Apologies for using an image, I don't know how to get the actual error code. Also not sure what I did wrong, but img tags are not working.)

Error screen: https://imgbb.com/z4jMn8z

Details that may be of use:

1. It's a bootable copy of Arch installed on a 32GB USB stick and running on a fairly new ASUS gaming laptop (which has windows installed on the primary HDD).
2. The error doesn't happen every time, seems to be intermittent
3. Windows runs fine without any errors, which I guess suggests its not a hardware fault?
4. The only error I can think of that I have been getting previous to this is that sometimes (particularly after a pacman upgrade) when I shut it down, it says that its powering off, but doesn't actually power off and I have to power it off manually.

Would really appreciate some help.

Thanks

Last edited by mrconfused (2024-01-31 08:13:34)

loqs · 2024-01-31 08:55:25

You can find the error using dmesg or journalctl. Please post the full output of system journal or dmesg to a pastebin.

seth · 2024-01-31 09:16:07

There's probably more error messages before this?

3. Windows runs fine without any errors, which I guess suggests its not a hardware fault?

3rd link below. Mandatory.
Disable it (it's NOT the BIOS setting!) and reboot windows and linux twice for voodo reasons.

it says that its powering off, but doesn't actually power off and I have to power it off manually.

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'

What if you just leave the system alone for a while - it might want to write a lot of data and USB keys are rather slow at that, notable when they age, because…

Arch installed on a 32GB USB stick

PSA: USB keys are not cheap SSDs.
If you treat them like that, they'll die in no time.
https://wiki.archlinux.org/title/Instal … ble_medium
https://wiki.archlinux.org/title/Improv … ads/writes

----

img tags are not working

You'd have to use the actual image, not the imgbb webpage, but you're not supposed to embed large images anyway - linking them is perfectly correct.

mrconfused · 2024-02-01 06:34:22

Please post the full output of system journal or dmesg to a pastebin

I have tried looking for the boot errors with journalctl, but I don't see them anywhere. Looking at my list of boots (journalctl --list-boots) Its looks like the failed boots haven't even been recorded. I'm new to using journalctl, so perhaps I'm missing something, but I have not been able to recover any of the errors. I was going to post an example log of one of my last boots but pastebin is currently under maintainence and there are no errors in it anyway.

There's probably more error messages before this?

I am now getting different errors on different boot attempts. Sometimes it gets as far as some arbitrary point (e.g. 'loading ramdisk') then it just freezes completely and I have to manually power off. On the rare occasion it boots successfully, it works fine without any other errors or crashing.

3rd link below. Mandatory.
Disable it (it's NOT the BIOS setting!) and reboot windows and linux twice for voodo reasons.

Thank you, I was not aware of this, I have changed it.

PSA: USB keys are not cheap SSDs.
If you treat them like that, they'll die in no time.

I also didn't know this, I will get an external SSD for this purpose at some point.

UPDATE:

Ok, i thought it might be my usb stick that was dying, so I copied the install to another usb stick and tried again. At first it worked, but on my next boot I began getting the same kind of errors again. As I mentioned above I have been unable to recover any of the errors from my logs. As far as I can tell, it doesn't get far along enough in the boot process to record the errors.

Also, as above, the errors are not the same every time. Sometimes I get a pageful of stuff like the one in my first post, sometimes it just hangs.

I'm beginning to worry its a hardware failure, but if that is the case:
1. Why does windows work without errors?
2. Why did it break specifically after my last pacman system upgrade?

Very much appreciate the help, thanks

seth · 2024-02-01 09:01:37

What was in the last update and have you tried the LTS kernel behavior?

Head_on_a_Stick · 2024-02-01 16:47:41

mrconfused wrote:

i thought it might be my usb stick that was dying, so I copied the install to another usb stick and tried again

If some files were corrupted because the original USB stick has problems then you have copied over the corrupted files to the new USB stick, which won't help.

Try a clean install instead.

mrconfused · 2024-02-02 03:37:00

Ok, I'm now 99% sure my computer has quite simply kicked the bucket.

This is what happened when I booted into the (latest version) of the arch installation medium:

https://ibb.co/THPWGcr

And there it hang.

Windows still works however.

Ok so now what? It looks like I'm either going to get my laptop repaired or else just buy a new computer.

Before I commit to either of those things, what else should I try to do? Given that windows still works, can I run some tests which might identify the exact problem?

I really appreciate the help you've given me, if I were doing this on my own I'd be going mad.

Any last advice? Things to do or not do next time I install arch? I really like arch, and would like to get better at using it.

Thanks

seth · 2024-02-02 09:18:38

grml.org - does that boot/run fine (you can also try eg. knoppix)

There's an increasing amount of page errors recently, errors when decompressing the initramfs (this started even before the 6.7 kernels) and at least isolated successes to boot w/ maxcpus=1, somewhat pointing at an at least SMP-related issue.
I don't believe that everyones PCs just recently fell apart.

https://bbs.archlinux.org/viewtopic.php … 1#p2140211

256 · 2024-02-02 17:25:51

In general, speaking from personal experience, "the hardware is broken" should be your last resort for explaining a problem (especially if it works on a different OS).

mrconfused · 2024-02-03 07:23:10

UPDATE:

So after reading your advice, I've done a few things:

First, I ran some rudimentary checks in windows, including a full memory diagnostic scan and there were no errors.

grml.org - does that boot/run fine (you can also try eg. knoppix)

Second, Grml from liveboot USB boots fine, and seems to work ok, except that my external keyboard isn't working. (this might signify something: see below)

This seems to suggest that (if it isn't hardware) there is a problem with arch which arose only during my second-to-last upgrade (I just do: pacman -Syu) AND the same problem is also to be found in the arch installation medium ISO itself.

BUT #1: I tried running a backup arch install that hadn't been updated for a month or more, and it booted up ok, but seemed to be running really slowly, and kept freezing.

BUT #2, there was another strange error (shortly before the error I took a screenshot of in my last post), while I was using the arch installation medium iso liveboot to begin a fresh arch install: When the 'select arch install medium' screen came up immediately after boot, my external keyboard worked fine, but once I actually got into the liveboot itself, it stopped working, and only the laptop keyboard worked. I unplugged and replugged the keyboard and it worked for a second, then stopped working again. The keyboard works fine everywhere else. (except grml)

So in summary memory seems to be fine. Windows and grml run, although not perfectly. Could there be some other problem with my hardware that might explain these issues? BIOS or motherboard fault or something?

If the issue is with arch itself, surely at least some other people would be having the same problem?

In general, speaking from personal experience, "the hardware is broken" should be your last resort for explaining a problem (especially if it works on a different OS).

Absolutely, I dont want to go spending money on new hardware if I dont have to. So what else can I do? I am woefully unqualified to investigate the problem on my own.

Last edited by mrconfused (2024-02-03 07:36:15)

seth · 2024-02-03 09:30:15

HoaS wrote:

If some files were corrupted because the original USB stick has problems then you have copied over the corrupted files to the new USB stick, which won't help.

seth wrote:

There's an increasing amount of page errors recently, errors when decompressing the initramfs (this started even before the 6.7 kernels) and at least isolated successes to boot w/ maxcpus=1, somewhat pointing at an at least SMP-related issue.

seth wrote:

What was in the last update and have you tried the LTS kernel behavior?

System integrity:

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'

And I'd suggest to switch the initramfs compression algorithm.

mrconfused · 2024-02-04 08:34:14

(Apologies for not getting to some of these things earlier)

seth wrote:

HoaS wrote:
If some files were corrupted because the original USB stick has problems then you have copied over the corrupted files to the new USB stick, which won't help.

That copy was another USB install that I tried. The arch installation medium ISO I used (which crashed, as above) was downloaded and written straight to a new USB with dd.

seth wrote:

There's an increasing amount of page errors recently, errors when decompressing the initramfs (this started even before the 6.7 kernels) and at least isolated successes to boot w/ maxcpus=1, somewhat pointing at an at least SMP-related issue.

I don't really know anything about page errors or SMP, so I don't know what this means. I will try read up on it a bit.

seth wrote:

What was in the last update ... ?

Here are the contents of my last few upgrades: (I *think* it was the first upgrade on the 30th that broke arch, or at least coincided with arch breaking, for me)

https://pastebin.com/GhUyNAKN

seth wrote:

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'

Here are the results of the above system integrity check:

https://ibb.co/K9KvgPg

seth wrote:

... have you tried the LTS kernel behavior? ... And I'd suggest to switch the initramfs compression algorithm

OK, so I installed and tried the linux LTS kernel (for the first time) and used the fallback initramfs (This last part is what you meant by switching the initramfs compression algorithm right?)

So it's booting! And has booted a few times in a row so far. This feels like progress, although the problems I've been having are intermittent so I hesitate to say its fixed. Also have gotten a couple of odd, transient graphical artifacts.

So what does this result mean? And what else can I do to test whether everything is working again?

Thank you again for all your help, it makes a huge difference.

seth · 2024-02-04 09:42:33

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files' > /tmp/foo.txt
cat /tmp/foo.txt | curl -F 'file=@-' 0x0.st

(You could also just pipe it directly but the query can take some time and you might timeout against the server)

The update has the kernel, mesa, systemd and xz but iff the LTS kernel works it's gonna be the kernel update.

(This last part is what you meant by switching the initramfs compression algorithm right?)

No.
https://man.archlinux.org/man/core/mkin … #VARIABLES

But the fallback kernel has pretty much all modules in the initramfs and doesn't have to load them from the disk - w/ your specific root device situation, you could test the behavior of the main kernel w/ its fallback initramfs.

mrconfused · 2024-02-05 08:59:04

seth wrote:

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files' > /tmp/foo.txt
cat /tmp/foo.txt | curl -F 'file=@-' 0x0.st

Ok so if I understood you correctly, you wanted me to run this on the working (LTS) fallback initramfs kernel and compare it to the same for the main kernel. This is the result I got from the LTS kernel:

https://0x0.st/Hkp0.txt

BUT: While running this I also got a few warnings which I have also copied here:

https://0x0.st/HkpG.txt

seth wrote:

(This last part is what you meant by switching the initramfs compression algorithm right?)
https://man.archlinux.org/man/core/mkin … #VARIABLES

I see what you mean now. If you want me to try changing this does it matter which compression type I choose, or does it only have to be different from the default (which I believe is zstd)?

seth wrote:

But the fallback kernel has pretty much all modules in the initramfs and doesn't have to load them from the disk - w/ your specific root device situation, you could test the behavior of the main kernel w/ its fallback initramfs.

I tried to boot into the main kernel again (with the fallback initramfs), and this time it worked ok. I ran the same query as for the LTS kernel above and it produced exactly the same result: When I uploaded it to 0x0.st, it created the same link. The warnings were also the same.

p.s. I forgot to mention in my last post that in order to install the LTS kernel I had to run a system upgrade because the LTS package database was out of date. I don't know if that may have changed anything.

What tests should I try next?

seth · 2024-02-05 09:32:10

Kernel doesn't matter, it checks the integrity of installed packages and looks unsuspicious.

does it matter which compression type I choose

Try lz4 or cat (uncompressed, you need sufficient space on the /boot partition!) - but first see below (this is currently less likely the cause)

When I uploaded it to 0x0.st, it created the same link

Yup, 0x0.st deduplicates - see above.

So the crucial question right now is whether you can boot any kernel reliably w/ the fallback initramfs while either kernel fails w/ the regular initramfs.

mrconfused · 2024-02-06 08:32:48

seth wrote:

So the crucial question right now is whether you can boot any kernel reliably w/ the fallback initramfs while either kernel fails w/ the regular initramfs.

Ok, well, after days of trying to get it not to crash on boot, now when I want to replicate the error, it's no longer crashing. I restarted half a dozen times in a row without any errors.

However, I did manage to get a screenshot of one of the graphical artifacts I've been getting (which also occur intermittently).

This screenshot is of the laptop screen immediately before my login screen appears. (And by login screen, I mean terminal, not X server) and the artifact lasts maybe half a second.

https://ibb.co/t8pPxJQ

I have never had this happen before.

Also, I now seem to be getting a possibly related problem in windows: Sometimes, at random, while not doing anything in particular, my screen will just suddenly go blank for a few seconds, and then will came back on. It's not the monitor.

I guess maybe these things suggest a problem with my GPU?

What else can I do to test this, or how can I force it to replicate the crash I was getting before? In spite of booting successfully several times in a row, I certainly don't think its fixed, I'm just waiting for it to happen again...

seth · 2024-02-06 09:47:44

I guess maybe these things suggest a problem with my GPU?

Could be a powersaving thing, check the journal for error related to drm or the VGA module - I don't think it's related to your booting errors, though.

What else can I do to test this, or how can I force it to replicate the crash I was getting before?

If all the non-fallback initramfs boots fine, you might be facing random corruptions of the initramfs - re-generating them might help to trigger this.

mrconfused · 2024-02-07 09:02:00

seth wrote:

Could be a powersaving thing, check the journal for error related to drm or the VGA module - I don't think it's related to your booting errors, though.

Couldn't find any references to either in the journal.

If all the non-fallback initramfs boots fine, you might be facing random corruptions of the initramfs - re-generating them might help to trigger this.

I regenerated the main linux and linux fallback images and rebooted. Still no errors.

Man this is frustrating, I just want to know what's broken so I can replace it.

Any more ideas?

I will try rebooting/regenerating a few more times, see if I can provoke the error. I might try running the arch installation medium ISO again, since that crashed when I tried it the first time, although I don't know what we can learn from that.

As always, appreciate the help.

Arch Linux

#1 2024-01-31 07:51:45

Arch error on start up

#2 2024-01-31 08:55:25

Re: Arch error on start up

#3 2024-01-31 09:16:07

Re: Arch error on start up

#4 2024-02-01 06:34:22

Re: Arch error on start up

#5 2024-02-01 09:01:37

Re: Arch error on start up

#6 2024-02-01 16:47:41

Re: Arch error on start up

#7 2024-02-02 03:37:00

Re: Arch error on start up

#8 2024-02-02 09:18:38

Re: Arch error on start up

#9 2024-02-02 17:25:51

Re: Arch error on start up

#10 2024-02-03 07:23:10

Re: Arch error on start up

#11 2024-02-03 09:30:15

Re: Arch error on start up

#12 2024-02-04 08:34:14

Re: Arch error on start up

#13 2024-02-04 09:42:33

Re: Arch error on start up

#14 2024-02-05 08:59:04

Re: Arch error on start up

#15 2024-02-05 09:32:10

Re: Arch error on start up

#16 2024-02-06 08:32:48

Re: Arch error on start up

#17 2024-02-06 09:47:44

Re: Arch error on start up

#18 2024-02-07 09:02:00

Re: Arch error on start up

Board footer