[SOLVED] The entire system is broken after a system upgrade

AndroGR · 2022-10-30 20:21:53

I went to upgrade my system casually (Everything was working fine before) today. After rebooting to use the new stuff, basically everything is broken: No sound, no network connection, everything related to GUI takes ages to load, USB drives not recognized. For some reason SDDM remains unaffected. Nothing suspicious in the pacman logs, the upgrade completed as expected, and definitely no damaged hardware as everything (Except the sound, which I couldn't test) seemed fine too. I installed another kernel along with the two I was using (vanilla and lts), rerun pacman -Syu, attempted downgrade, nothing of success.

Last edited by AndroGR (2022-10-31 20:14:39)

xerxes_ · 2022-10-30 20:33:44

What kernel version you had before update and to what version did you update (6.0.5 or 6.0.6) ?

Post full output of commands: 'sudo journalctl -b' and 'lspci -nnk'.

seth · 2022-10-30 20:36:18

Compare "uname -a" and "pacman -Qs linux"

AndroGR · 2022-10-30 21:50:20

xerxes_ wrote:

What kernel version you had before update and to what version did you update (6.0.5 or 6.0.6) ?
Post full output of commands: 'sudo journalctl -b' and 'lspci -nnk'.

I ran 6.0.2 and upgraded to 6.0.5.
For the first command, I couldn't fit the entire output, I assume it's enough though: https://pastebin.com/W70gSLAm

The second one right here: https://pastebin.com/Ey04rbU9

AndroGR · 2022-10-30 21:54:24

I understand you guys are trying to help, but if possible, don't ask me to post commands with huge output. It's a pretty hard job to move the output from my computer to the mobile, then copy the entire thing from termux to Firefox.

jasonwryan · 2022-10-30 22:03:24

AndroGR wrote:

I understand you guys are trying to help, but if possible, don't ask me to post commands with huge output. It's a pretty hard job to move the output from my computer to the mobile, then copy the entire thing from termux to Firefox.

This is one line...

seth wrote:

Compare "uname -a" and "pacman -Qs linux"

walkingstickfan · 2022-10-30 22:03:43

My recommendation would be to install and use another Linux distro on the same machine, using the same username and password (helpful if you can't access the other distro(s) at all by mounting their filesystem under a file manager or via terminal), and use a shared data partition. That way, you'll have another Linux distro to fall back on if you encounter this situation again and you can save the output to the shared data partition; no mobile required. ;-)

Last edited by walkingstickfan (2022-10-30 22:05:08)

AndroGR · 2022-10-30 22:08:51

jasonwryan wrote:

AndroGR wrote:
I understand you guys are trying to help, but if possible, don't ask me to post commands with huge output. It's a pretty hard job to move the output from my computer to the mobile, then copy the entire thing from termux to Firefox.
This is one line...
seth wrote:
Compare "uname -a" and "pacman -Qs linux"

I wasn't referring to any specific line, I was just saying in general that it's not an easy job.

AndroGR · 2022-10-30 22:10:42

walkingstickfan wrote:

My recommendation would be to install and use another Linux distro on the same machine, using the same username and password (helpful if you can't access the other distro(s) at all by mounting their filesystem under a file manager or via terminal), and use a shared data partition. That way, you'll have another Linux distro to fall back on if you encounter this situation again and you can save the output to the shared data partition; no mobile required. ;-)

I have two partitions already, a home partition and a root partition. Well, actually it's four, including the UEFI and swap ones. And also, I have some pretty important stuff in there.

loqs · 2022-10-30 22:21:23

Can you try 6.0.3 from the ALA see if that has the issue you could also install linux-lts 5.19.74 which should hopefully not have the issue and give you a working kernel.

cfr · 2022-10-30 23:52:10

Can you boot the lts kernel? You said you'd already got it installed in an earlier post.

virtualperson72 · 2022-10-31 05:16:08

Fellow AMD/Radeon user here. My system also became unusable around kernel 6.0.2 after a pacman patch last Thursday or Friday. Many kernel kaudit warnings/messages at boot up (hadn't seen those before problems started), no network (interfaces would not come up, wireguard would not initialize, long (perhaps infinite) delays on iptables setup scripts) no graphics, could not open LUKS partitions, system often did not finish the boot up cycle: logging in was difficult. Occasionally would get network service when booting the LTS kernel (but not always (it has also been recently updated)).

Other much older non AMD/Radeon systems are working fine with kernel 6.0.5.

Possible other pages that seem to describe this:
https://www.reddit.com/r/linux_gaming/c … ely_60_is/
https://gitlab.freedesktop.org/drm/amd/ … te_1581119
https://www.reddit.com/r/linux_gaming/c … ntext=1000

I was able to get things going again by booting to the LTS kernel (several times), finally getting a network connection, then downgrading all packages back to October 10, 2022 by updating my /etc/pacman.d/mirrorlist as described in the ALA section 3.2.

System now works as it should (network, wireguard, iptables, LUKS, bootup, all good). Sending good karma to developers in the hopes they figure this one out.

seth · 2022-10-31 07:17:00

From https://pastebin.com/W70gSLAm the OP seems to run 6.0.5 - the posted data is however mostly a mess and barely usable.

It's a pretty hard job to move the output from my computer to the mobile, then copy the entire thing from termux to Firefox.

Boot the bad system, shut down NOT using the power button and then boot a live distro (arch install iso, grml, …) and access the journal of the installed system, https://wiki.archlinux.org/title/System … al_to_view
Then upload it from the live distro (even if you just have a console, 1st link below)
If a somewhat clean shutdown isn't an option, you can also try to dump the journal into a file

sudo journalctl -b > ~/bad_journal.txt; sudo sync

and upload that from the live distro.

Ftr, OP claims

I ran 6.0.2 and upgraded to 6.0.5.

and at least suggests that the LTS kernel is likewise affected, so the links in #12 are probably not relevant.

AndroGR · 2022-10-31 11:21:08

seth wrote:

From https://pastebin.com/W70gSLAm the OP seems to run 6.0.5 - the posted data is however mostly a mess and barely usable.
It's a pretty hard job to move the output from my computer to the mobile, then copy the entire thing from termux to Firefox.
Boot the bad system, shut down NOT using the power button and then boot a live distro (arch install iso, grml, …) and access the journal of the installed system, https://wiki.archlinux.org/title/System … al_to_view
Then upload it from the live distro (even if you just have a console, 1st link below)
If a somewhat clean shutdown isn't an option, you can also try to dump the journal into a file
sudo journalctl -b > ~/bad_journal.txt; sudo sync
and upload that from the live distro.
Ftr, OP claims
I ran 6.0.2 and upgraded to 6.0.5.
and at least suggests that the LTS kernel is likewise affected, so the links in #12 are probably not relevant.

Just to let everybody know, all kernels are having the exact same issue and show the same problem here. I imagine it's some sort of patch applied lately, and they also applied it to all kernels.

Last edited by AndroGR (2022-10-31 11:30:34)

AndroGR · 2022-10-31 11:22:00

seth wrote:

From https://pastebin.com/W70gSLAm the OP seems to run 6.0.5 - the posted data is however mostly a mess and barely usable.
It's a pretty hard job to move the output from my computer to the mobile, then copy the entire thing from termux to Firefox.
Boot the bad system, shut down NOT using the power button and then boot a live distro (arch install iso, grml, …) and access the journal of the installed system, https://wiki.archlinux.org/title/System … al_to_view
Then upload it from the live distro (even if you just have a console, 1st link below)
If a somewhat clean shutdown isn't an option, you can also try to dump the journal into a file
sudo journalctl -b > ~/bad_journal.txt; sudo sync
and upload that from the live distro.
Ftr, OP claims
I ran 6.0.2 and upgraded to 6.0.5.
and at least suggests that the LTS kernel is likewise affected, so the links in #12 are probably not relevant.

Where do I upload the journal, without a GUI? that's exactly the problem I have.

AndroGR · 2022-10-31 11:26:23

loqs wrote:

Can you try 6.0.3 from the ALA see if that has the issue you could also install linux-lts 5.19.74 which should hopefully not have the issue and give you a working kernel.

I already tried downgrading, no results. I believe it's a commit even before 6.0.2, as only the LTS kernel is usable.

I also tried downgrading the LTS (Which happened to be working at least fine enough for the most part) but I ended up stucking in the boot process, so definitely not a good idea either.

AndroGR · 2022-10-31 11:28:34

FYI, it looks like Qt applications are completely unaffected by this (Except KDE Plasma). Reminds me of a previous mesa bug, which would only happen on Gnome and GTK. Perhaps that's a hint?

seth · 2022-10-31 12:44:23

Stop bumping, if nobody replied, edit your previous post. See the forum rules itr.

AndroGR wrote:

Where do I upload the journal, without a GUI? that's exactly the problem I have.

seth wrote:

Then upload it from the live distro (even if you just have a console, 1st link below)

HaroldMorgan · 2022-10-31 15:07:31

Same problem when updating from 6.0.5. kernel to 6.0.6

"uname -a" and "pacman -Qs linux" give different results (6.0.5 and 6.0.6 accordingly)

Video:
VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Wani [Radeon R5/R6/R7 Graphics]

CPU:
AMD A10-9600P RADEON R5, 10 COMPUTE CORES 4C+6G

After update iwctl stopped working (spamming No HMAC(SHA1) support found and such messages in jorunal) and Xorg stopped working too (couldn't find working display)

So it seems that I'm sitting at 6.0.5 kernel at least for now

UPDATE: This was my own mistake of not correctly mounting boot partition to /boot on new Arch install, which was causing this behavior, you can ignore this message

Last edited by HaroldMorgan (2022-10-31 15:57:10)

Scimmia · 2022-10-31 15:09:43

All of that sounds like a configuration issue. Usually that you don't have the /boot partition mounted when you update.

HaroldMorgan · 2022-10-31 15:56:42

Scimmia wrote:

All of that sounds like a configuration issue. Usually that you don't have the /boot partition mounted when you update.

Yeah, turns out in my case this actually was a configuration issue of not mounting boot partition correctly, noted that down in original message. Thanks for pointing out!

AndroGR · 2022-10-31 16:50:35

HaroldMorgan wrote:

Same problem when updating from 6.0.5. kernel to 6.0.6
"uname -a" and "pacman -Qs linux" give different results (6.0.5 and 6.0.6 accordingly)

Video:
VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Wani [Radeon R5/R6/R7 Graphics]
CPU:
AMD A10-9600P RADEON R5, 10 COMPUTE CORES 4C+6G

After update iwctl stopped working (spamming No HMAC(SHA1) support found and such messages in jorunal) and Xorg stopped working too (couldn't find working display)
So it seems that I'm sitting at 6.0.5 kernel at least for now
UPDATE: This was my own mistake of not correctly mounting boot partition to /boot on new Arch install, which was causing this behavior, you can ignore this message

Also replying to the original question which asked for the output of these commands, no differences here: Both kernels print as "6.0.5-arch1-1". So I imagine not related to my issue.

AndroGR · 2022-10-31 17:00:38

Alright, I managed to get my hands on a pretty old kernel (5.19.11-arch1-1). One strange thing I noticed is that it wanted me to regenerate my grub configuration (sudo grub-mkconfig -o /boot/grub/grub.cfg).

loqs · 2022-10-31 17:21:42

If you have rasdaemon enabled try disabling it [1]. Please try 6.0.3.arch3-1. Edit your last post if no one else has posted so you do not bump the thread.

[1] https://bugs.archlinux.org/task/76354

Last edited by loqs (2022-10-31 17:21:54)

AndroGR · 2022-10-31 17:30:20

loqs wrote:

If you have rasdaemon enabled try disabling it [1]. Please try 6.0.3.arch3-1. Edit your last post if no one else has posted so you do not bump the thread.
[1] https://bugs.archlinux.org/task/76354

There is no '6.0.3.arch3-1' for me, only 6.0.3.arch2. I am gonna try that as I suppose nothing changed.
E1: Nevermind, I found the arch3-1 version.
E2: Yep, disabling rasdaemon and switching to this kernel works. Including all problems going away.

Last edited by AndroGR (2022-10-31 17:40:20)

Arch Linux

#1 2022-10-30 20:21:53

[SOLVED] The entire system is broken after a system upgrade

#2 2022-10-30 20:33:44

Re: [SOLVED] The entire system is broken after a system upgrade

#3 2022-10-30 20:36:18

Re: [SOLVED] The entire system is broken after a system upgrade

#4 2022-10-30 21:50:20

Re: [SOLVED] The entire system is broken after a system upgrade

#5 2022-10-30 21:54:24

Re: [SOLVED] The entire system is broken after a system upgrade

#6 2022-10-30 22:03:24

Re: [SOLVED] The entire system is broken after a system upgrade

#7 2022-10-30 22:03:43

Re: [SOLVED] The entire system is broken after a system upgrade

#8 2022-10-30 22:08:51

Re: [SOLVED] The entire system is broken after a system upgrade

#9 2022-10-30 22:10:42

Re: [SOLVED] The entire system is broken after a system upgrade

#10 2022-10-30 22:21:23

Re: [SOLVED] The entire system is broken after a system upgrade

#11 2022-10-30 23:52:10

Re: [SOLVED] The entire system is broken after a system upgrade

#12 2022-10-31 05:16:08

Re: [SOLVED] The entire system is broken after a system upgrade

#13 2022-10-31 07:17:00

Re: [SOLVED] The entire system is broken after a system upgrade

#14 2022-10-31 11:21:08

Re: [SOLVED] The entire system is broken after a system upgrade

#15 2022-10-31 11:22:00

Re: [SOLVED] The entire system is broken after a system upgrade

#16 2022-10-31 11:26:23

Re: [SOLVED] The entire system is broken after a system upgrade

#17 2022-10-31 11:28:34

Re: [SOLVED] The entire system is broken after a system upgrade

#18 2022-10-31 12:44:23

Re: [SOLVED] The entire system is broken after a system upgrade

#19 2022-10-31 15:07:31

Re: [SOLVED] The entire system is broken after a system upgrade

#20 2022-10-31 15:09:43

Re: [SOLVED] The entire system is broken after a system upgrade

#21 2022-10-31 15:56:42

Re: [SOLVED] The entire system is broken after a system upgrade

#22 2022-10-31 16:50:35

Re: [SOLVED] The entire system is broken after a system upgrade

#23 2022-10-31 17:00:38

Re: [SOLVED] The entire system is broken after a system upgrade

#24 2022-10-31 17:21:42

Re: [SOLVED] The entire system is broken after a system upgrade

#25 2022-10-31 17:30:20

Re: [SOLVED] The entire system is broken after a system upgrade

Board footer