You are not logged in.

#1 2023-03-04 10:36:29

joaot
Member
Registered: 2023-03-01
Posts: 14

After routine pacman -Syu update, Xorg mostly frozen when waking up

Hello,

I installed Arch around 2 years ago on a decently-supported Lenovo T480 with an Nvidia GPU.

Things were fine until a recent 'sudo pacman -Syu'. This updated Xorg, Nvidia drivers and Linux kernel (at least presumably,  I don't remember the full list).

I use Xorg with 'startx' from my regular user and the rather thin '~/.xinitrc' ends with 'exec openbox-session', starting up Openbox.

Since then, I observe the following:

* Whenever I come out of 'slock' after waking up the laptop, the window manager is _almost_ frozen.  The mouse pointer works and moves around but clicks don't take effect. The application having focus is still running and I can interact with it using some of the keys.  If it's a video window, for example, video will play fine.  But I can't control Openbox to switch windows or launch new ones.

* My custom  build  of "Simple Terminal" 'st' hangs once in a while.  This also didn't happen before.  I've recompiled it and it sill happens

The only way out I have is to switch to different TTY, log in, and 'killall xinit'.

Where should I look for suspicious messages logs these problems?  I've looked at '~/.local/share/xorg/Xorg.0.log' and 'dmesg'.  They each have some suspicious lines but I don't know what I'm supposed to look for.

I wonder if there's somewhere in Arch where old pre-update Linux kernels are stored.  I could try booting into one.

João

PS: This is my first forum post. I read the rules, did some research, but no solution, so I though I'd post.  Let me know if you need more information, and thanks in advance.

Offline

#2 2023-03-04 15:43:19

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 68,411

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

I've looked at '~/.local/share/xorg/Xorg.0.log' and 'dmesg'. They each have some suspicious lines but I don't know what I'm supposed to look for.

"Problems" wink
Post them (resp. rather the system journal than dmesg)


From the symptoms, see whether this is indeed the S3 and skip slock.
Then try whether slock alone can trigger this.

The only way out I have is to switch to different TTY, log in, and 'killall xinit'.

Not necessarily:
/etc/X11/xorg.conf.d/10-server.conf

Section "ServerFlags"
    Option         "DontZap" "false"
    Option         "DontVTSwitch" "false"
    Option         "AllowDeactivateGrabs" "true" # also run setxkbmap -option grab:break_actions - Ctrl+Alt+Keypad-Divide
    Option         "AllowClosedownGrabs" "true" # Ctrl+Alt+Keypad-Multiply
EndSection

nb. that both Allow*Grabs options are security risks and only for debugging.
If a client grabs the input, they'll forcefully release the grab resp. kill the offending client (mind you that this could be openbox!)

If you

sleep 60; xdotool key "XF86LogGrabInfo"

then trigger the problem and wait until xdotool fires, you'll also get the (active) grabs printed into the xorg log (obviously don't cut off the grab before)

Ftr

the rather thin '~/.xinitrc'

See the last link below on what to include at least.

I wonder if there's somewhere in Arch where old pre-update Linux kernels are stored.  I could try booting into one.

You can downgrade the kernel from the pacman cache or the https://wiki.archlinux.org/title/Arch_Linux_Archive
You'll also have to downgrade the nvidia driver to keep it aligned w/ the kernel.

Maybe the lts kernel can narrow the issue.

Offline

#3 2023-03-05 14:07:26

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

Thanks very much for your reply and all the suggestions Seth,

So here's what I've done so far, which narrows down the search space a bit:

1. I've downgraded my Linux kernel to 6.1.11 using

sudo pacman -U file://linux-6.1.11.arch1-1-x86_64.pkg.tar.zst file://linux-headers-6.1.11.arch1-1-x86_64.pkg.tar.zst

as described in the Wiki.  Then rebooted. 6.1.11 is the kernel I was using before this started.  Since this didn't help, I'm cautiously ruling out a kernel regression.

2. I've tried multiple times to reproduce from just 'slock'. and the problem does _not_ occur. 

So now a bit of extra info.  Here's my '~/.xinitrc'  My '~/.Xresources' file is just the single line: 'Emacs*useXIM: false'.  I don't think the Conky setup matters, as it happens regardless of Conky.

From the symptoms, see whether this is indeed the S3 and skip slock.

S3 = suspend, right?  Yes, I do believe it's suspend-specific.  I normally suspend with just `sustemctl suspend`.

Not necessarily: /etc/X11/xorg.conf.d/10-server.conf

Thanks.  I actually knew about that.  But I don't want to turn it on since I might forget to turn it off :-)

sleep 60; xdotool key "XF86LogGrabInfo"

This is interesting.  But unfortunately I don't have a 100% repro to trigger the bug.  I happens often enough though, so I'll give it a shot.  Please confirm if this is your idea:

1. invoke this command
2. then suspend
3. then wake up, witness the hang
4.wait for xdotool to fire
5. keep xinit alive,  switch to a TTY
6. save the '~/.local/share/xorg/Xorg.0.log'. 
7. save the system journal (will search about how to do that)

Wonder how 'sleep' behaves across 'systemctl suspend', but I guess I'll find out.

Offline

#4 2023-03-05 14:21:25

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 68,411

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

Please confirm if this is your idea:

Except for "7" ("sudo journalctl -b -<n>" provides access to older journals, increase the "<n>" digit to go back in time)
I'd also have you try to trigger this by suspending without slock being invoked.

About your xinitrc:
1. it's broken, see the last linke below and the note on what to include at least
2.

xinput set-prop 'TPPS/2 IBM TrackPoint' "libinput Accel Speed" 1
xinput --disable 'SynPS/2 Synaptics TouchPad'
xinput --disable 'Synaptics TM3276-022'

has me somewhat uneasy.
Why do you disable those devices and let's assume you'd not: would that break your input?
Try to use a config like https://man.archlinux.org/man/extra/xor … SS_SECTION and https://man.archlinux.org/man/extra/xor … #Option~40 to block them.
What could happen is that the device gets (virtually) re-attached by the suspend and not disabled and interferes w/ your input.

Offline

#5 2023-03-05 14:32:11

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

Murphy's law dictates that it's hard to reproduce when I want it to. But It'll happen soon enough.  I'm now always sleeping like this

(sleep 20 && xdotool key "XF86LogGrabInfo" && echo 'go check it out!')&; systemctl suspend

Offline

#6 2023-03-07 10:16:34

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

What could happen is that the device gets (virtually) re-attached by the suspend and not disabled and interferes w/ your input.

I don't know how to answer because I don't understand the concepts of attachment/suspension/interference, but I think I added those lines is to disable the trackpad for power-saving reasons.  But I took them out.

I also added "the last if block of /etc/X11/xinit/xinitrc" to my own ~/.xinitrc (curiosity: if this is mandatory, why isn't it also automatic?).  I don't fully understand what the two scripts in /etc/X11/xinit/xinitrc.d are doing (40-libcanberra-gtk-module.sh, 50-systemd-user.sh) , but they don't look harmful either.

I've reproduced the situation once again now, but I didn't have the opportunity to collect the logs.  Will happen soon enough hopefully.

Offline

#7 2023-03-07 12:02:45

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 68,411

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

curiosity: if this is mandatory, why isn't it also automatic?

What do you mean by "automatic"? It's in the global xinitrc and there's no "automatic" local one.
xinit won't do any of that because it's relevant to systemd/logind, not xinit.

they don't look harmful either

The important one is the one that imports some environment variables into the session.

Offline

#8 2023-03-07 22:03:59

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

What do you mean by "automatic"

If an xinitrc without those things is indeed "broken" then it should be harder to break it. .  Some programs (emacs, sbcl,...) use a site config and a user config and the former is included before unless you issue certain flags to the invocation. But it's way beyond this thread to discuss this tangent.

Anyway, good (bad) news!.  It happened again: and I got the logs from the X session that is barely operable (as I described earlier), after waiting some seconds for the 'sleep 60' that you suggested ends.

http://0x0.st/Hi-0.txt /var/log/Xorg.log.0.old EDIT!: I updated the link because before I had provided ~/.local/share/xinit/Xorg.log.0, which is not what you want I think.  Id doesn't seem to contain the output of the grab command, for example.

http://0x0.st/Hi-4.txt: System journal (since boot 'sudo journalctl -b') is what I used.

http://0x0.st/Hi-y.txt Dmesg output (for good measure).

I also have another clue, may be of help.  There's another regression that I didn't describe in the original post that seems to be related to this.  My 'st' terminal sometimes hangs, doesn't redisplay, a hard hang.  I have to kill the process, killing the window doesn't work. I'll try to attach a GDB to it one of these times.  Anyway, terminals are cheap so when it happens, I make another one and continue working. Now, curiously, it seems that is in these X sessions where this happened that the _next_ suspend will cause the big freeze I described.

Last edited by joaot (2023-03-07 22:14:35)

Offline

#9 2023-03-07 22:15:13

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 24,759

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

Get rid of xf86-video-intel, and the config snippet that loads it. It hasn't seen any notable developments in 10 years and is detrimental on systems newer than sandybridge for the majority of usecases.

Offline

#10 2023-03-08 00:16:38

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

Get rid of xf86-video-intel, and the config snippet that loads it. It hasn't seen any notable developments in 10 years and is detrimental on systems newer than sandybridge for the majority of usecases.

Thanks, but detrimental in what way? Crash-inducing detrimental? Anyway I removed it and replaced it with this alternative

Offline

#11 2023-03-08 07:03:26

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 68,411

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

There're no active grabs in that log, "detrimental" in terms of framebuffer corruptions and also you've an optimus system and

[    58.662] (**) |-->Input Device "Keyboard0"
[    58.662] (**) |-->Input Device "Mouse0"

reeks of nvidia-settings generated nonsense.
Not sure whether it's relevant, but your wifi connection flickers after resuming from the S3.

If an xinitrc without those things is indeed "broken" then it should be harder to break it

No, you "should" read the wiki.

Offline

#12 2023-03-08 08:26:34

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

There're no active grabs in that log,

I don't follow.  Should I have done something more other than those confirmed 7 steps? Should I have
Did you find any clues related to the hang I described in the original post that didn't happen before the `pacman -Syu`

reeks of nvidia-settings generated nonsense.

OK. I took out some those sections of '/etc/X11/xorg.conf/15-nvidia.conf'.  As far as i remember, I just installed nvidia packages from the repo.

Not sure whether it's relevant, but your wifi connection flickers after resuming from the S3.

I don't notice it. Should I embark on fixing that too?

Offline

#13 2023-03-08 08:56:05

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

For those familiar with Xlib internals, this is summarized backtrace of occasional 'st' hang that will eventually lead to the hang on the next suspend.

This never happened before the 'pacman -Syu' update. It seems to happen when I type quickly, but not always.

Happy to dig into those frames and examine variable values, etc.

#0  0x00007fca8cf5f9a4 in __GI___poll (fds=fds@entry=0x7ffea0d1af68, nfds=nfds@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fca8ce4526b in poll (__timeout=-1, __nfds=1, __fds=0x7ffea0d1af68) at /usr/include/bits/poll2.h:39
#2  _xcb_conn_wait (c=c@entry=0x55d2d0d44e40, vector=vector@entry=0x0, count=count@entry=0x0, cond=<optimized out>) at /usr/src/debug/libxcb/libxcb-1.15/src/xcb_conn.c:508
#3  0x00007fca8ce46d1d in _xcb_conn_wait (count=0x0, vector=0x0, cond=0x55d2d0d44e80, c=0x55d2d0d44e40) at /usr/src/debug/libxcb/libxcb-1.15/src/xcb_conn.c:474
#4  xcb_wait_for_event (c=0x55d2d0d44e40) at /usr/src/debug/libxcb/libxcb-1.15/src/xcb_in.c:703
#5  0x00007fca8d1d7b09 in _XReadEvents (dpy=0x55d2d0d42990) at /usr/src/debug/libx11/libX11-1.8.4/src/xcb_io.c:482
#6  _XReadEvents (dpy=0x55d2d0d42990) at /usr/src/debug/libx11/libX11-1.8.4/src/xcb_io.c:454
#7  0x00007fca8d1be43a in XIfEvent (dpy=0x55d2d0d42990, event=0x7ffea0d1b1a0, predicate=0x7fca8d1fa490 <_CheckCMEvent>, arg=0x55d2d0df9800 "\300 -\215\312\177")
    at /usr/src/debug/libx11/libX11-1.8.4/src/IfEvent.c:73
#8  0x00007fca8d204f10 in _XimXRead (im=0x55d2d0df9800, recv_buf=0x7ffea0d1b5d0 "", buf_len=2048, ret_len=0x7ffea0d1b2d4) at ../modules/im/ximcp/imTrX.c:477
#9  0x00007fca8d1fc523 in _XimReadData (im=im@entry=0x55d2d0df9800, len=len@entry=0x7ffea0d1b346, buf=buf@entry=0x7ffea0d1b5d0 "", buf_size=buf_size@entry=2048)
    at ../modules/im/ximcp/imTransR.c:165
#10 0x00007fca8d20547c in _XimRead (im=0x55d2d0df9800, len=0x7ffea0d1b3de, buf=0x7ffea0d1b5d0 "", buf_size=2048, predicate=0x7fca8d1f0bf0 <_XimSetICValuesCheck>,
    arg=0x55d2d0e00630 "") at ../modules/im/ximcp/imTransR.c:235
#11 0x00007fca8d1f5d6c in _XimProtoSetICValues (xic=0x55d2d0e00630, arg=<optimized out>) at ../modules/im/ximcp/imDefIc.c:779
#12 0x00007fca8d1e0c5f in XSetICValues (ic=0x55d2d0e00630) at xlibi18n/ICWrap.c:336
#13 0x000055d2d088afd4 in xximspot (x=<optimized out>, y=<optimized out>) at x.c:1697
#14 0x000055d2d0887951 in draw () at st.c:2736
#15 0x000055d2d088a2cf in run () at x.c:2010
#16 0x000055d2d088b7da in main (argc=<optimized out>, argv=<optimized out>) at x.c:2095

Offline

#14 2023-03-08 12:55:01

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 68,411

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

I don't follow.

It just means that the symptoms are not down to an active grab.

As far as i remember, I just installed nvidia packages from the repo.

The source doesn't matter. nvidiasettings and nvidia-xconfig don't produce very useful config files.

Should I embark on fixing that too?

It might be interesting to see whether the problem remains when you're completely offline (disconnect before the sleep and connect explicitly after)

Happy to dig into those frames and examine variable values, etc.

The backtrace shows st polling the server for events, there's nothing outright special about that.

Offline

#15 2023-03-09 12:43:45

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

It might be interesting to see whether the problem remains when you're completely offline (disconnect before the sleep and connect explicitly after)

It remains.  Here is Xorg log http://0x0.st/Hict.txt and journalctl log http://0x0.st/Hicv.txt

The backtrace shows st polling the server for events, there's nothing outright special about that.

The special thing about it is that it shows (part of) the hung state of the process, which is always a prerequisite for the next wake-up-after-sleep to cause the hang described.  So inspecting that state and figuring out why the server isn't replying could be useful.

Anyway, I've just used the Arch Linux Archive to roll back to about a month ago and things look fine.

I'm almost sure the problem is gone because another problem is now gone. Before my rollback,  invoking 'conky' with this configuration file:

conky.config = {
   -- own_window_type = 'desktop'
}

conky.text = [[
${color grey}Info:$color ${scroll 32 Conky $conky_version - $sysname $nodename $kernel $machine}
]]

Lead to an X-related error output by conky (which I've not in front of me at the moment, unless conky writes some log). Anyway I had to uncomment that own_window_type line to get conky to work. 

Now, after the rollback, the original configuration file works again.

Offline

#16 2023-03-09 21:31:44

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 68,411

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

https://github.com/brndnmtthws/conky/issues/1443 ?

So inspecting that state and figuring out why the server isn't replying could be useful.

Either because there're no events to report or for a different reason, but you're not learning that from the client.

Coming back to the original symptoms:
alter your xinitrc to instead of

exec openbox-session
openbox-session
while true; do
   sleep 1 # protection against perma-crashing "openbox" hogging your system
   openbox
done

Suspend, resume, if it's frozen, kill openbox. New openbox should start, see whether things are now "unfrozen".

Offline

#17 2023-03-10 16:48:56

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

Yes, seems like it.  Seems like a separate conky bug though. As a test partial-pgraded the conky package to the latest and it breaks with this otherwise functioning system.  Pity, as it would have made a good test.

> Either because there're no events to report or for a different reason, but you're not learning that from the client.

The client is dynamically linking Xlib (which was updated along many thing)  A bug there or in one of its dependencies could lead to the problem.

Suspend, resume, if it's frozen, kill openbox. New openbox should start, see whether things are now "unfrozen".

Maybe I'll try that. But as I noted, it only happens when 'st' suddenly hangs. And that only happens sometimes.

I think more promising safer strategy is to use the archlinux archive and bisect.  I'm know that I'm now at 11-Feb-2023 and presumably in a "good" state. Upgrading to 26-Feb-2023 is a "bad" state. Seems like 4 tests should be enough.  I'll move to 19-Feb-2023 soon and keep reporting here.

Offline

#18 2023-03-14 01:12:53

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

I  think more promising safer strategy is to use the archlinux archive and bisect.  I'm know that I'm now at 11-Feb-2023 and presumably in a "good" state. Upgrading to 26-Feb-2023 is a "bad" state. Seems like 4 tests should be enough.  I'll move to 19-Feb-2023 soon and keep reporting here.

19-Feb-2023 looking good.  Will move to 23-Feb-2023 soon and keep reporting here.

Offline

#19 2023-03-14 12:25:39

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

19-Feb-2023 looking good.  Will move to 23-Feb-2023 soon and keep reporting here.

Moved to 23-Feb-2023 got a 'st' lockup after about half an hour.  Moving to 21-Feb-2023.  Gotta love a good bisect.

Last edited by joaot (2023-03-14 12:25:59)

Offline

#20 2023-03-14 13:31:48

joaot
Member
Registered: 2023-03-01
Posts: 14

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

Moved to 23-Feb-2023 got a 'st' lockup after about half an hour.  Moving to 21-Feb-2023.

Also locks up, after some basic terminal usage. Moved to 20-Feb-2023, no lockups so far.  I'm eyeing 21-Feb as the first "bad" day. If 20-Feb is good, these  are the downgrades that fixed it:

[2023-03-14T12:48:42+0000] [ALPM] downgraded alsa-card-profiles (1:0.3.66-2 -> 1:0.3.65-6)
[2023-03-14T12:48:42+0000] [ALPM] downgraded curl (7.88.1-1 -> 7.88.0-4)
[2023-03-14T12:48:43+0000] [ALPM] downgraded gnutls (3.8.0-1 -> 3.7.9-1)
[2023-03-14T12:48:43+0000] [ALPM] downgraded mesa (22.3.5-1 -> 22.3.4-1)
[2023-03-14T12:48:43+0000] [ALPM] downgraded freerdp (2:2.10.0-1 -> 2:2.9.0-1)
[2023-03-14T12:48:43+0000] [ALPM] downgraded libibus (1.5.28-1 -> 1.5.27-1)
[2023-03-14T12:48:43+0000] [ALPM] downgraded ibus (1.5.28-1 -> 1.5.27-1)
[2023-03-14T12:48:43+0000] [ALPM] downgraded iproute2 (6.2.0-1 -> 6.1.0-5)
[2023-03-14T12:48:43+0000] [ALPM] downgraded libnm (1.42.0-2 -> 1.42.0-1)
[2023-03-14T12:48:43+0000] [ALPM] downgraded mariadb-libs (10.11.2-1 -> 10.10.3-1)
[2023-03-14T12:48:44+0000] [ALPM] downgraded networkmanager (1.42.0-2 -> 1.42.0-1)
[2023-03-14T12:48:44+0000] [ALPM] downgraded python-pydantic (1.10.5-1 -> 1.10.4-1)
[2023-03-14T12:48:44+0000] [ALPM] downgraded python-setuptools (1:67.3.2-1 -> 1:67.3.0-1)

Offline

#21 2023-03-14 13:41:12

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 68,411

Re: After routine pacman -Syu update, Xorg mostly frozen when waking up

mesa and (lib)ibus are yelling at you…

Offline

Board footer

Powered by FluxBB