[SOLVED] Upgraded and suddenly most commands are stalled

pelegs · 2023-12-12 13:45:23

Yesterday I upgraded (i.e. running pacman -Syyu).
This also installed a new kernel (6.6.3 iirc).

I use only i3 for interface because I don't like working with the mouse.
Anyway, I noticed that i3 didn't load the status-bar, but I couldn't fix it.

While trying to understand what's wrong, I noticed that many commands just stall completely: ping, pacman, shutdown, ... - all simply give a black output and even ctrl+c and ctrl+q don't unhang them.
Curiously some programs do work, e.g. neovim, zathura and htop. I can't make sense of any of that.
I used an archiso usb to chroot into my system and re-install the kernel - everything is the same.

Any ideas how to proceed and/or how to even start finding the issue?

Last edited by pelegs (2023-12-13 13:57:46)

V1del · 2023-12-12 14:09:09

Upgrade the system including the kernel to 6.6.6 which contains a fix for a critical bug in the network subsystem that got introduced in 6.6.5.

Last edited by V1del (2023-12-12 14:10:44)

pelegs · 2023-12-12 14:15:49

I can't use pacman (it's one of the programs that get stuck).
Can I do that via the live usb? Should I chroot again, or is there a way to update my system directly?

V1del · 2023-12-12 14:18:42

Since a chroot will use the kernel and network configuration of the ISO, you should be able to invoke pacman as normal after the chroot -- make sure your partitions are logically mounted correctly before the chroot, especially if you have a seperate /boot partition (mount the root partition first and an existing boot partition to /mnt/boot if that's where you mounted your root to).

pelegs · 2023-12-12 14:24:39

alright, I will try this. Thanks!

pelegs · 2023-12-12 15:22:24

After upgrading I get the following error:

[FAILED] Failed to mount /boot
[DEPEND] Dependency failed for Local File Systems.
You are in emergency mode.

(there's more, but it seems like a well-known message)

I tried to login with my root password, but since my keyboard is a bit misbehaving, it's rather difficult to enter the right password.
Is there anything to do via chroot? Maybe I can temporarily change my root password to something easy to type, and then use that.
I will update tomorrow if I still have issues.

Last edited by pelegs (2023-12-12 15:22:34)

Scimmia · 2023-12-12 15:23:32

You probably forgot to mount your /boot partition. I don't know how, as he was very clear about not forgetting it, but here we are.

Last edited by Scimmia (2023-12-12 15:24:18)

pelegs · 2023-12-12 15:57:28

Yeah, I thought I did exactly that, but I guess I didn't.

The thing is, I have three separate partitions: nvme0n1p1, nvme01p2 and nvme01p3. It seems that p2 is the root, so I mounted it to /mnt. p1 is the boot (it has vmlinuz, grub, etc.). So I mounted it on /mnt/boot. p3 is my home folder so it's irrelevant.

Was I wrong in any of the above? I'm not exactly an expert.

Last edited by pelegs (2023-12-12 16:09:37)

Scimmia · 2023-12-12 17:20:28

No, that should do it, as long as you mounted both before you chrooted.

Check your fstab, make sure things are what you think they are. See if you have a vmlinuz file in the /boot dir on your root partition, and use `file` to check what version it is. Check the one on nvme0n1p1 as well.

pelegs · 2023-12-12 17:38:54

ok, I will do that tomorrow (as I'm not near the computer until then).

seth · 2023-12-12 20:54:22

Also check the kernel command line in a journal from the installes system - if the kernel shows in as "/boot/vmlinuz-linux", you're actually booting from the root partition.

pelegs · 2023-12-13 06:46:34

ok, so -

1. I checked /etc/fstab. It is exactly what I was expecting: /dev/nvme0n1p2 is mounted on / (ext4 filesystem). /dev/nvme0n1p1 is mounted on /boot (vfat filesystem). nvme0n1p3 is mounted on /home.
2. Output of /proc/cmdline:

BOOT_IMAGE=/vmlinuz-linux root=UUID=xxx[identical to that of /dev/nvme0n1p1] rw zswap.enabled=0 rootfstype=ext4 loglevel=3 quiet

It all seems ok according to what I expect and what seth wrote.

Last edited by pelegs (2023-12-13 06:47:58)

seth · 2023-12-13 08:23:03

Do't obfuscate the UUID

identical to that of /dev/nvme0n1p1

It seems that p2 is the root, so I mounted it to /mnt. p1 is the boot

That doesn't fit.
Please post the output of "lsblk -f", your fstab and a system journal from the installed system.
You can paste stuff from the commandline using the tip in the 1st link below.

pelegs · 2023-12-13 09:08:15

Content of "/etc/fstab": https://0x0.st/H3nL.txt
Output of "lsblk -f": https://0x0.st/H3nf.txt
Output of "journalctl": https://0x0.st/H3nO.txt

Last edited by pelegs (2023-12-13 09:08:29)

pelegs · 2023-12-13 12:39:41

So I found this old thread: https://bbs.archlinux.org/viewtopic.php?id=269588, in which a very similar problem to mine is presented and you helped solve the problem. I indeed had the exact signs you mentioned in your first comment, so I followed the same steps described.

I ran pacman -S linux, and now have the 6.6.6 version of the kernel installed. When I comment out the boot partition from "/etc/fstab" I actually get all the way to my login manager after a long (LONG) wait. During the long wait, the following written on the screen:

/dev/nvme0n1p2: clean, 268184/1310720 files, 2939815/5242880 blocks

...then after a while more errors show up, but I can't capture them before getting into the login manager.
When it does eventually get to the login manager, everything is super slow and both the mouse pad and usb devices don't seem to work. I guess some modules aren't loaded? Anyway, I can access tty1 at least.

I also booted using "fallback initramfs" mode, and (in tty1) ran "journalctl -xb". The output shows that there's a time out waiting for something to load:

Dec 13 13:18:49 hostcomp systemd[1]: dev-zram0.device: Job dev-zram0.device/start times out.
Dec 13 13:18:49 hostcomp systemd[1]: Times out waiting for device /dev/zram0.

The full log is available here: https://0x0.st/H357.txt
(the second log message says that it uses the old kernel: "Linux version 6.6.5-arch1-1")

So all in all, I'm awfully confused.

V1del · 2023-12-13 12:53:19

From the usb stick.

mount /dev/nvme0n1p2 /mnt
file /mnt/boot/vmlinuz-linux
mount /dev/nvme0n1p1 /mnt/boot
file mnt/boot/vmlinuz-linux

which bootloader are you using?

seth · 2023-12-13 12:53:50

Dec 13 09:35:10 hostname kernel: Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=c4ec70ea-d1f0-40d2-b35a-a87dc62a7bd3 rw zswap.enabled=0 rootfstype=ext4 loglevel=3 quiet

actually has the correct root partition (equals nvme0n1p2) and shows you're booting from a boot partition, so if you followed https://bbs.archlinux.org/viewtopic.php?id=269588 and installed the kernel w/o mounting the boot partition, you predictably end up w/ booting the old kernel.

=> Fix that, make sure to mount the /boot partition and re-install the kernel.
Then boot the system (still w/o the boot entry in the fstab to skip that obstacle) and run and post the outputs of

ls /lib/modules /usr/lib/modules
modinfo vfat
modinfo zram
sudo modprobe -v vfat
sudo modprobe -v zram

Ftr, when posting journals please *never* use "-x", it's effectively spam.

pelegs · 2023-12-13 13:38:16

@V1del: before I mount "/dev/nvme0n1p1" I get that the kernel is version 6.6.6-arch1-1.

After I mount it I get that the kernel is version 6.6.5-arch1-1.
So yeah, that kind of makes sense to me with regards to what seth wrote.

@Seth:

=> Fix that, make sure to mount the /boot partition and re-install the kernel.

I have a feeling I simply don't understand how to do that.
At the moment I have "/boot" commented out in "/etc/fstab".
If I do "mount /dev/nvme0n1p1 /boot" I get the following error:

mount: /boot: unknown filesystem type 'vfat'.

Did you mean mounting and installing via chroot?

seth · 2023-12-13 13:42:36

You need to boot the intall iso and fix this from there, yes.
Mount the root partition to /mnt, clear the /mnt/boot path, then mount the /boot partition to /mnt/boot and re-install the kernel (you can chroot or use pacman --sysroot /mnt for this)
Then try to reboot the intalled system and run the commands in #17

pelegs · 2023-12-13 13:57:33

ok, this was the thing that confused me... so I needed to copy /boot from the liveusb to my system's /boot and then install the new kernel.
This actually solved all my issues it seems - I got back all functionality including all the stuff i3 was missing, firefox working, internet automatically loading etc.
If you're still interested, I can run the commands you asked in #17, but maybe it's a bit redundant now. Your decision :-)

I will mark the thread with "[SOLVED]", and later today sum up the steps to solve in a comment.

Many, many, MANY thanks! Also for your patience!

Last edited by pelegs (2023-12-13 13:58:30)

Scimmia · 2023-12-13 14:01:02

No, you DO NOT copy anything from the live usb. You literally just make sure everything is mounted correctly and reinstall the kernel. The fact that the new kernel image is on the root partition proves that nvme0n1p1 was NOT mounted to /boot inside the chroot when you installed the kernel.

pelegs · 2023-12-13 14:03:32

ok, I need to learn more about these things it seems.

seth · 2023-12-13 14:28:41

You certainly installed 6.6.6 into the wrong location because you mis-applied the situation of the other thread to yourself.
That doesn't explain the previously broken 6.6.5 installation, though, except if you had attempted an update before post #14 - yuor pacman log can tell.

1. Makre sure to restore the fstab entry for the /boot partition, otherwise the next update will get you back to square one
2. Because you copied the archiso /boot, please post your complete system journal for the boot:

sudo journalctl -b | curl -F 'file=@-' 0x0.st

and the output of

ls -lR /boot

so we can ensure
a) you're not just booting the installation initramfs
b) you didn't copy a whole bunch of stuff that has no business on your intalled systems /boot directory

pelegs · 2023-12-13 16:14:37

1. Done.
2. a. http://0x0.st/H37t.txt
b. http://0x0.st/H37v.txt

Again, thanks

seth · 2023-12-13 16:23:00

Looks sane enough.

Arch Linux

#1 2023-12-12 13:45:23

[SOLVED] Upgraded and suddenly most commands are stalled

#2 2023-12-12 14:09:09

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#3 2023-12-12 14:15:49

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#4 2023-12-12 14:18:42

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#5 2023-12-12 14:24:39

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#6 2023-12-12 15:22:24

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#7 2023-12-12 15:23:32

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#8 2023-12-12 15:57:28

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#9 2023-12-12 17:20:28

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#10 2023-12-12 17:38:54

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#11 2023-12-12 20:54:22

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#12 2023-12-13 06:46:34

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#13 2023-12-13 08:23:03

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#14 2023-12-13 09:08:15

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#15 2023-12-13 12:39:41

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#16 2023-12-13 12:53:19

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#17 2023-12-13 12:53:50

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#18 2023-12-13 13:38:16

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#19 2023-12-13 13:42:36

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#20 2023-12-13 13:57:33

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#21 2023-12-13 14:01:02

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#22 2023-12-13 14:03:32

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#23 2023-12-13 14:28:41

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#24 2023-12-13 16:14:37

Re: [SOLVED] Upgraded and suddenly most commands are stalled

#25 2023-12-13 16:23:00

Re: [SOLVED] Upgraded and suddenly most commands are stalled

Board footer