Terminal freezes with majority of commands [SOLVED]

darkeve · 2021-10-27 00:59:11

Hi guys,

So I have not made any major changes to my arch linux system but it seems have a strange issue where apart from basic commands like ls. Any command entered causes the terminal to freeze so I am forced to shut down the terminal, not even sudo shutdown now works. Wondering what steps I can do to find the root cause.

Though I had an issue with the machine just hanging when the shutdown command was done so I turned it off forcefully with the power button. Rarely the terminal does work without an issue but the power command rarely works as well as a seperate issue.

I am running Arch 5.14.14-arch1-1 using the awesome Windows manager, zsh 5.8 and it is running on a X1 carbon 7th gen.

What an output of df -h looks like

Filesystem Use mounted
Dev 0% /dev
Run 1% /run
/dev/mapper/volgroup-lv_root 46% /
tmpfs 0% /dev/shm
tmpfs 1% /tmp
/dev/mapper/volgroup-lv_home 11% /home
tmpfs 1% /run/user/1000
tmpfs 0% /run/user/0

Last edited by darkeve (2021-10-29 08:11:06)

Trilby · 2021-10-27 01:01:28

Does this happen in the tty too? Or with other shells (e.g., bash)? Are you using oh-my-zsh?

darkeve · 2021-10-27 01:13:05

Yes it does, tried with bash and the same issue and I am.using oh my zsh too.

Trilby · 2021-10-27 01:45:01

Ah ... wait, are you sure it happens with bash. If it does, that's a hell of a coincidence as I predicted you were using OMZ and that this would be the source of the problem (because OMZ serves no purpose other than to cause ridiculous problems like this). But bash does not use OMZ ... unless of course you are launching bash from within a zsh session that already loaded OMZ.

In either case, try removing OMZ and see if the problem goes away. Alternatively, try with a different user that has not had OMZ configured. If you log in as root in a tty do you see this problem (and please tell me OMZ isn't configured for the root account).

Last edited by Trilby (2021-10-27 01:45:34)

seth · 2021-10-27 06:58:20

apart from basic commands like ls. Any command entered

What is "any command" specifically?
Does ctrl+q "unfreeze" the shell?
If you do this in a GUI, does anything else in the session "hang"?
https://wiki.archlinux.org/title/S.M.A.R.T.#smartctl - esp. "smartctl -a", in doubt from a live distro (grml)

darkeve · 2021-10-27 23:06:58

Trilby wrote:

Ah ... wait, are you sure it happens with bash. If it does, that's a hell of a coincidence as I predicted you were using OMZ and that this would be the source of the problem (because OMZ serves no purpose other than to cause ridiculous problems like this). But bash does not use OMZ ... unless of course you are launching bash from within a zsh session that already loaded OMZ.
In either case, try removing OMZ and see if the problem goes away. Alternatively, try with a different user that has not had OMZ configured. If you log in as root in a tty do you see this problem (and please tell me OMZ isn't configured for the root account).

I uninstalled ohmyzsh and rebooted the machine when the terminal decided to work and i still have the same issue so it has nothing to do with that. Root is using bash and I also logged into tty with my root account and still the same issue. I should also mention it seems my wlan0 device is down when this terminal issue does happen but when the terminal decides to work fine on a boot then the wlan0 device is fine too and is normally connected to my Internet via wireless.

darkeve · 2021-10-27 23:13:05

seth wrote:

apart from basic commands like ls. Any command entered
What is "any command" specifically?
Does ctrl+q "unfreeze" the shell?
If you do this in a GUI, does anything else in the session "hang"?
https://wiki.archlinux.org/title/S.M.A.R.T.#smartctl - esp. "smartctl -a", in doubt from a live distro (grml)

Its strange, commands like ping, ls, vim doesn't freeze the terminal but commands like trying to get my protonvpn command running freezes or sudo su. So it's quite random.

Ctrl+q doesn't seem to bring the terminal out of its frozen state and I can't seem to get anything to run to test if anything hangs. Though if I run top in a different terminal that still seems fine after the other terminal freezes.

I'll look into that smartctl command.

darkeve · 2021-10-27 23:30:56

I ran a journalctl n -50 command after trying to run a command that would freeze the terminal and this was the output.

https://imgur.com/a/BUvtEoW

loqs · 2021-10-27 23:43:43

After a terminal hangs and you switch terminal can you run ps and check on the shell and commands state i.e. if the shell was bash and the command sudo

ps l -C bash
ps l -C sudo

darkeve · 2021-10-28 00:29:11

loqs wrote:

After a terminal hangs and you switch terminal can you run ps and check on the shell and commands state i.e. if the shell was bash and the command sudo
ps l -C bash
ps l -C sudo

So I first ran sudo su in my zsh shell and got this after running those commands

https://imgur.com/a/rTzK5Yx

Then I closed the terminal and tried to switch to bash but I got this

https://imgur.com/a/WTZH6th

So I logged into root via tty which is running bash and got this

https://imgur.com/a/VW93PxV

loqs · 2021-10-28 00:59:15

As you can not use sudo, login as root and execute the following to show what is blocking the tasks in the kernel:

echo w > /proc/sysrq-trigger

Then post the output of dmesg. See the tip box from pastebin to post the output to a pastebin, rather than posting images of text.

seth · 2021-10-28 07:19:46

commands like trying to get my protonvpn command running freezes or sudo su. So it's quite random.

So far everything points sudo…

1. can you just "su"? ("sudo su" is a bad pattern anyway, it's "sudo -i" or "su")
2. when you sudo anything, do you get to enter your creds?
3.

type sudo; pacman -Qikk sudo

4. can you "sudo -l"? "sudo -V"?
5. after making some terminals freeze with "random" commands, do NOT close those terminals and dump the output of "ps aux"

darkeve · 2021-10-28 19:05:33

seth wrote:

commands like trying to get my protonvpn command running freezes or sudo su. So it's quite random.
So far everything points sudo…
1. can you just "su"? ("sudo su" is a bad pattern anyway, it's "sudo -i" or "su")
2. when you sudo anything, do you get to enter your creds?
3.
type sudo; pacman -Qikk sudo
4. can you "sudo -l"? "sudo -V"?
5. after making some terminals freeze with "random" commands, do NOT close those terminals and dump the output of "ps aux"

Su allows me to log into root but i don't think its a sudo problem as for example running ip addr freezes the terminal in both my user and root account. But then again using the su command to get into root. The shutdown now command seems to process without sudo.

2. No I don't get to see my credit apart from typing su

3. http://dpaste.com//5ECDGWZBQ

4. Nope, freezes the terminal

5. http://dpaste.com//ABF4UJ87J

seth · 2021-10-28 19:11:36

You've quite some network managing services running… and most are waiting for IO (possibly on each other)

hostnamectl
find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Trilby · 2021-10-28 19:17:33

How about `strace sudo -V` - you may want to direct the stderr output to a file so if / when the terminal freezes, you can still get the output from the contents of the file. E.g.:

strace sudo -V 2>strace.log

Then post the content of "strace.log"

darkeve · 2021-10-28 19:21:30

loqs wrote:

As you can not use sudo, login as root and execute the following to show what is blocking the tasks in the kernel:
echo w > /proc/sysrq-trigger
Then post the output of dmesg. See the tip box from pastebin to post the output to a pastebin, rather than posting images of text.

Caused 3 terminals to freeze

http://dpaste.com//2STAUJZHD

darkeve · 2021-10-28 19:23:44

seth wrote:

You've quite some network managing services running… and most are waiting for IO (possibly on each other)
hostnamectl
find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Should mention on the boot where the terminal doesn't decide to work it seems network manager fails to start too but on the boots where the terminal works then there are no issues with the network manager if that means anything.

seth · 2021-10-28 19:27:17

The dmesg already reveals that the stalls are in cfg80211 and netlink_*
This is by far the most malign outcome of concurrent network managers I've seen so far…

Please post the output of the requested commands.

darkeve · 2021-10-28 19:45:12

seth wrote:

You've quite some network managing services running… and most are waiting for IO (possibly on each other)
hostnamectl
find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Hostctl

http://dpaste.com//CRHC2FCST

Command only works when the terminal works

2nd command, when the terminal works

http://dpaste.com//4YW33ZGDD

When the terminal doesn't work. In case it gives a different output

http://dpaste.com//AFZPTFRVN

darkeve · 2021-10-28 20:00:48

Trilby wrote:

How about `strace sudo -V` - you may want to direct the stderr output to a file so if / when the terminal freezes, you can still get the output from the contents of the file. E.g.:
strace sudo -V 2>strace.log
Then post the content of "strace.log"

http://dpaste.com//9MU69WE4Y

seth · 2021-10-28 20:01:22

You've a static hostname (good) but run systemd-networkd, networkmanager (+wpa_supplicant) and iwd concurrently.
If you want to eg. go with NetworkManager, disable iwd and the systemd-networkd* services & sockets, reboot and see what happens.

darkeve · 2021-10-28 20:20:11

seth wrote:

You've a static hostname (good) but run systemd-networkd, networkmanager (+wpa_supplicant) and iwd concurrently.
If you want to eg. go with NetworkManager, disable iwd and the systemd-networkd* services & sockets, reboot and see what happens.

So when I stop iwd or network manager or systemd-networkd it seems to cause my connection to drop and wlan0 device is not known. I use nmcli to connect to my WiFi.

darkeve · 2021-10-28 20:24:47

darkeve wrote:

seth wrote:
You've a static hostname (good) but run systemd-networkd, networkmanager (+wpa_supplicant) and iwd concurrently.
If you want to eg. go with NetworkManager, disable iwd and the systemd-networkd* services & sockets, reboot and see what happens.
So when I stop iwd or network manager or systemd-networkd it seems to cause my connection to drop and wlan0 device is not known. I use nmcli to connect to my WiFi.

Actually nevermind previous comment. Rebooting with your recommendation still seems like the network is fine with just networkmanager running but I'll reboot a few times to see if the terminal still breaks.

darkeve · 2021-10-28 21:00:42

seth wrote:

You've a static hostname (good) but run systemd-networkd, networkmanager (+wpa_supplicant) and iwd concurrently.
If you want to eg. go with NetworkManager, disable iwd and the systemd-networkd* services & sockets, reboot and see what happens.

I've rebooted quite a few times now. Seems the issue is fixed and the laptop shutdowns properly everytime too. Many thanks.

I don't understand how it suddenly started having these issues as I'm pretty sure I've been running the same setup for a while. How strange how it breaks the terminal like that.

seth · 2021-10-29 03:01:11

sudo looks up the hostname, the resolver will try to use the network. The impact on "ip" is obvious.
Usually the network just fails or constantly toggles, as mentioned: afair this is the first deadlock I've seen for this condition.

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

Arch Linux

#1 2021-10-27 00:59:11

Terminal freezes with majority of commands [SOLVED]

#2 2021-10-27 01:01:28

Re: Terminal freezes with majority of commands [SOLVED]

#3 2021-10-27 01:13:05

Re: Terminal freezes with majority of commands [SOLVED]

#4 2021-10-27 01:45:01

Re: Terminal freezes with majority of commands [SOLVED]

#5 2021-10-27 06:58:20

Re: Terminal freezes with majority of commands [SOLVED]

#6 2021-10-27 23:06:58

Re: Terminal freezes with majority of commands [SOLVED]

#7 2021-10-27 23:13:05

Re: Terminal freezes with majority of commands [SOLVED]

#8 2021-10-27 23:30:56

Re: Terminal freezes with majority of commands [SOLVED]

#9 2021-10-27 23:43:43

Re: Terminal freezes with majority of commands [SOLVED]

#10 2021-10-28 00:29:11

Re: Terminal freezes with majority of commands [SOLVED]

#11 2021-10-28 00:59:15

Re: Terminal freezes with majority of commands [SOLVED]

#12 2021-10-28 07:19:46

Re: Terminal freezes with majority of commands [SOLVED]

#13 2021-10-28 19:05:33

Re: Terminal freezes with majority of commands [SOLVED]

#14 2021-10-28 19:11:36

Re: Terminal freezes with majority of commands [SOLVED]

#15 2021-10-28 19:17:33

Re: Terminal freezes with majority of commands [SOLVED]

#16 2021-10-28 19:21:30

Re: Terminal freezes with majority of commands [SOLVED]

#17 2021-10-28 19:23:44

Re: Terminal freezes with majority of commands [SOLVED]

#18 2021-10-28 19:27:17

Re: Terminal freezes with majority of commands [SOLVED]

#19 2021-10-28 19:45:12

Re: Terminal freezes with majority of commands [SOLVED]

#20 2021-10-28 20:00:48

Re: Terminal freezes with majority of commands [SOLVED]

#21 2021-10-28 20:01:22

Re: Terminal freezes with majority of commands [SOLVED]

#22 2021-10-28 20:20:11

Re: Terminal freezes with majority of commands [SOLVED]

#23 2021-10-28 20:24:47

Re: Terminal freezes with majority of commands [SOLVED]

#24 2021-10-28 21:00:42

Re: Terminal freezes with majority of commands [SOLVED]

#25 2021-10-29 03:01:11

Re: Terminal freezes with majority of commands [SOLVED]

Board footer