Arch Linux

loqs

Member

Registered: 2014-03-06

Posts: 18,045

Re: thinkpad crashes while idle - power management issue

Some record keeping

git bisect bad
87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb is the first bad commit
commit 87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb
Author: Linyu Yuan <quic_linyyuan@quicinc.com>
Date:   Mon Apr 25 18:18:05 2022 +0800

    usb: typec: ucsi: add a common function ucsi_unregister_connectors()
    
    In error path of ucsi_init(), it will unregister all valid ucsi connectors,
    and similar operation also happen in ucsi_unregister(),
    add a common function ucsi_unregister_connectors() for two places,
    inside this function, if con->wq is NULL, it will break the loop,
    if other kind of error happen after con->wq allocated,
    ucsi/typec related API is safe to unregister.
    
    Also in ucsi_init(), it allocate number of (ucsi->cap.num_connectors + 1)
    connectors, there is one extra as the ending,
    ucsi_unregister_connectors() is safe to unregister all ucsi connectors
    according ucsi->cap.num_connectors,
    remove the extra one connector to save memory.
    
    Reviewed-by: Heikki Krogerus <heikki.krogerus@linux.intel.com>
    Signed-off-by: Linyu Yuan <quic_linyyuan@quicinc.com>
    Link: https://lore.kernel.org/r/1650881886-25530-2-git-send-email-quic_linyyuan@quicinc.com
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

 drivers/usb/typec/ucsi/ucsi.c | 53 +++++++++++++++++++++++--------------------
 1 file changed, 29 insertions(+), 24 deletions(-)

git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [3d7cb6b04c3f3115719235cc6866b10326de34cd] Linux 5.19
git bisect bad 3d7cb6b04c3f3115719235cc6866b10326de34cd
# status: waiting for good commit(s), bad commit known
# good: [4b0986a3613c92f4ec1bdc7f60ec66fea135991f] Linux 5.18
git bisect good 4b0986a3613c92f4ec1bdc7f60ec66fea135991f
# good: [c011dd537ffe47462051930413fed07dbdc80313] Merge tag 'arm-soc-5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect good c011dd537ffe47462051930413fed07dbdc80313
# good: [5d4af9c1f04ab0411ba5818baad9a68e87f33099] Merge branch 'mv88e6xxx-fixes-for-reading-serdes-state'
git bisect good 5d4af9c1f04ab0411ba5818baad9a68e87f33099
# bad: [7a68065eb9cd194cf03f135c9211eeb2d5c4c0a0] Merge tag 'gpio-fixes-for-v5.19-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux
git bisect bad 7a68065eb9cd194cf03f135c9211eeb2d5c4c0a0
# bad: [54c2cc79194c961a213c1d375fe3aa4165664cc4] Merge tag 'usb-5.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
git bisect bad 54c2cc79194c961a213c1d375fe3aa4165664cc4
# good: [6a31a95135da0bb2c5349e49e37d76e9909ab7ea] staging: r8188eu: remove include/rtw_debug.h
git bisect good 6a31a95135da0bb2c5349e49e37d76e9909ab7ea
# good: [04d93b2b8bc7a68ec45a6a156f34a611ede5aa60] Merge tag 'spdx-5.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/spdx
git bisect good 04d93b2b8bc7a68ec45a6a156f34a611ede5aa60
# good: [c069d2756c01ed36121fae6a42c14fdf1325c71d] serial: sifive: Sanitize CSIZE and c_iflag
git bisect good c069d2756c01ed36121fae6a42c14fdf1325c71d
# bad: [3120aac6d0ecd9accf56894aeac0e265f74d3d5a] usb: dwc2: gadget: don't reset gadget's driver->bus
git bisect bad 3120aac6d0ecd9accf56894aeac0e265f74d3d5a
# good: [0f01017191384e3962fa31520a9fd9846c3d352f] usb: dwc3: Don't switch OTG -> peripheral if extcon is present
git bisect good 0f01017191384e3962fa31520a9fd9846c3d352f
# bad: [7a20917d30fb627bebff6bacbe860ad4eb3a04d6] device property: Add helper to match multiple connections
git bisect bad 7a20917d30fb627bebff6bacbe860ad4eb3a04d6
# good: [61f879ab755b7dd6f468c55358a3061092682041] media: remove third argument of usb_maxpacket()
git bisect good 61f879ab755b7dd6f468c55358a3061092682041
# good: [bdddc253b0938a0063798881d1f6a971ea1d8943] usb: rework usb_maxpacket() using usb_pipe_endpoint()
git bisect good bdddc253b0938a0063798881d1f6a971ea1d8943
# good: [7a60fa06e80596dba5381fb018f5102ae4897658] dt-bindings: usb: renesas,usbhs: Document RZ/G2UL bindings
git bisect good 7a60fa06e80596dba5381fb018f5102ae4897658
# bad: [3c162511530c234f95091bdc7225f641e5f35090] usb: typec: ucsi: Wait for the USB role switches
git bisect bad 3c162511530c234f95091bdc7225f641e5f35090
# bad: [87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb] usb: typec: ucsi: add a common function ucsi_unregister_connectors()
git bisect bad 87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb
# first bad commit: [87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb] usb: typec: ucsi: add a common function ucsi_unregister_connectors()

KloolK

Member

Registered: 2022-08-24

Posts: 27

Re: thinkpad crashes while idle - power management issue

Doing a sanity check with: https://bbs.archlinux.org/viewtopic.php … 6#p2054086

confirmed good.

loqs

Member

Registered: 2014-03-06

Posts: 18,045

Re: thinkpad crashes while idle - power management issue

5.19.4 with 87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb reverted the config is still from 5.18.16 but I doubt that will matter for this test:
https://drive.google.com/file/d/1erJ7ML … sp=sharing linux-5.19.4-1.1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1c-2DQR … sp=sharing linux-headers-5.19.4-1.1-x86_64.pkg.tar.zst

KloolK

Member

Registered: 2022-08-24

Posts: 27

Re: thinkpad crashes while idle - power management issue

5.19.4 with 87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb reverted the config is still from 5.18.16 but I doubt that will matter for this test:
https://drive.google.com/file/d/1erJ7ML … sp=sharing linux-5.19.4-1.1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1c-2DQR … sp=sharing linux-headers-5.19.4-1.1-x86_64.pkg.tar.zst

Linux arch-esr-2399 5.19.4-1.1 #1 SMP PREEMPT_DYNAMIC Sat, 27 Aug 2022 18:30:22 +0000 x86_64 GNU/Linux

with 87d0e2f41b8cc2018499be4e8003fa8c09b6f2fb reverted

no issues.

loqs

Member

Registered: 2014-03-06

Posts: 18,045

Re: thinkpad crashes while idle - power management issue

Thank you for all the testing.  Please open a bug report upstream on https://bugzilla.kernel.org/buglist.cgi Product=Drivers Component=USB include the git bisect log and the dmesg from a kernel with the issue.

KloolK

Member

Registered: 2022-08-24

Posts: 27

Re: thinkpad crashes while idle - power management issue

Done: https://bugzilla.kernel.org/show_bug.cgi?id=216422

Thanks for your help!

ehaynes

Member

Registered: 2022-08-19

Posts: 9

Re: thinkpad crashes while idle - power management issue

Thank you both so much! I was too busy to follow along yesterday and today, but I really appreciate everything!

KloolK

Member

Registered: 2022-08-24

Posts: 27

Re: thinkpad crashes while idle - power management issue

You're welcome!

For anyone who wants to make sure it is the same issue, verify

that https://bbs.archlinux.org/viewtopic.php … 7#p2054097 is bad

and that https://bbs.archlinux.org/viewtopic.php … 2#p2054112 is good.

Last edited by KloolK (2022-08-28 12:52:30)

KloolK

Member

Registered: 2022-08-24

Posts: 27

Re: thinkpad crashes while idle - power management issue

The issue was already known and a fix (revert) is on its way: https://git.kernel.org/pub/scm/linux/ke … de4788826e

Avatar

Member

Registered: 2010-04-02

Posts: 31

Re: thinkpad crashes while idle - power management issue

Sorry to bump this thread, does anyone know the timeline of when this will be pushed to a stable release?
There have been a few stable releases since the above report, but it never seems to be included.

loqs

Member

Registered: 2014-03-06

Posts: 18,045

Re: thinkpad crashes while idle - power management issue

Sorry to bump this thread, does anyone know the timeline of when this will be pushed to a stable release?
There have been a few stable releases since the above report, but it never seems to be included.

5.19.8
https://git.kernel.org/pub/scm/linux/ke … 3a9bee6729

Avatar

Member

Registered: 2010-04-02

Posts: 31

Re: thinkpad crashes while idle - power management issue

5.19.8
https://git.kernel.org/pub/scm/linux/ke … 3a9bee6729

Thanks, looking forward to it.

loqs

Member

Registered: 2014-03-06

Posts: 18,045

Re: thinkpad crashes while idle - power management issue

5.19.8
https://git.kernel.org/pub/scm/linux/ke … 3a9bee6729

Thanks, looking forward to it.

5.19.8 is now in testing.  Did that resolve the issue?

Avatar

Member

Registered: 2010-04-02

Posts: 31

Re: thinkpad crashes while idle - power management issue

5.19.8 is now in testing.  Did that resolve the issue?

Oh great, I didn't know it would be so soon after I posted so in desperation I built the stable rc from https://git.kernel.org/pub/scm/linux/ke … nux-5.19.y
I've switched to 5.19.8 from testing now, but I'm very hopeful because my system ran without any issues for most of yesterday (with my custom kernel which should be almost identical)

I'll report back in a few hours if I see any hiccups.
How long does a package stay in testing before being moved to core?

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 23,203

Re: thinkpad crashes while idle - power management issue

Minor kernel releases normally not long, I'm surprised it even went to testing for a notable period of time, maybe a maintainer has noticed an actual issue on their own machine?

Avatar

Member

Registered: 2010-04-02

Posts: 31

Re: thinkpad crashes while idle - power management issue

I'll report back in a few hours if I see any hiccups.

No issues at all today, so I suspect it's fixed.
The real test for me is leaving my laptop idle for the night but I'm relatively confident that it's fixed now.

Last edited by Avatar (2022-09-09 12:13:57)

Avatar

Member

Registered: 2010-04-02

Posts: 31

Re: thinkpad crashes while idle - power management issue

The real test for me is leaving my laptop idle for the night but I'm relatively confident that it's fixed now.

Been a bit slack with following up, but working perfectly for me.
Would be good to see it moved to core.

boxuser007

Member

Registered: 2010-11-25

Posts: 4

Re: thinkpad crashes while idle - power management issue

Just to report another crash case on Lenovo T480 with same symptoms as author of this post. Crashes started when i upgraded kernel to 5.19.X.

My symptoms were:

• a blinking CAPS-LOCK button, but leaving Gnome Desktop working, even able to move mouse on screen, but Gnome didn't responded to click events. It crashed just in a middle of a work.

• Gnome wasn't able to display battery status - by investigating logs i saw that upower crashed. Wasn't able to open gnome-control-center, as it relied on upower.

  Sep 12 18:55:09 WBOX upowerd[970]: did not recognise type Unknown, please report
  Sep 12 18:55:09 WBOX kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
  Sep 12 18:55:09 WBOX kernel: #PF: supervisor instruction fetch in kernel mode
  Sep 12 18:55:09 WBOX kernel: #PF: error_code(0x0010) - not-present page
  Sep 12 18:55:09 WBOX kernel: PGD 0 P4D 0 
  Sep 12 18:55:09 WBOX kernel: Oops: 0010 [#1] PREEMPT SMP PTI
  Sep 12 18:55:09 WBOX kernel: CPU: 1 PID: 970 Comm: upowerd Tainted: G           OE     5.19.7-arch1-1 #1 e547dd5fb390dbeba7568a7a787b4abde4680315
  Sep 12 18:55:09 WBOX kernel: Hardware name: LENOVO :CENSORED:, BIOS :CENSORED: (1.36 ) 10/13/2020
  Sep 12 18:55:09 WBOX kernel: RIP: 0010:0x0
  Sep 12 18:55:09 WBOX kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
  Sep 12 18:55:09 WBOX kernel: RSP: 0018:ffffa7fc40f5bcb8 EFLAGS: 00010202
  Sep 12 18:55:09 WBOX kernel: RAX: 0000000000000000 RBX: ffffffff97b84d08 RCX: ffffffff97b84d08
  Sep 12 18:55:09 WBOX kernel: RDX: ffffa7fc40f5bcc0 RSI: 0000000000000004 RDI: ffff8b7c04622800
  Sep 12 18:55:09 WBOX kernel: RBP: 0000000000000004 R08: ffff8b7c04622838 R09: ffff8b7c0e4dd000
  Sep 12 18:55:09 WBOX kernel: R10: ffffa7fc40f5be20 R11: 0000000000000001 R12: ffff8b7c06519000
  Sep 12 18:55:09 WBOX kernel: R13: ffff8b7c04622838 R14: ffffffff97b84d08 R15: ffff8b7c04622800
  Sep 12 18:55:09 WBOX kernel: FS:  00007fde44fbd3c0(0000) GS:ffff8b7f6f480000(0000) knlGS:0000000000000000
  Sep 12 18:55:09 WBOX kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  Sep 12 18:55:09 WBOX kernel: CR2: ffffffffffffffd6 CR3: 0000000100b2e001 CR4: 00000000003706e0
  Sep 12 18:55:09 WBOX kernel: Call Trace:
  Sep 12 18:55:09 WBOX kernel:  <TASK>
  Sep 12 18:55:09 WBOX kernel:  power_supply_show_property+0xbf/0x240
  Sep 12 18:55:09 WBOX kernel:  dev_attr_show+0x19/0x40
  Sep 12 18:55:09 WBOX kernel:  sysfs_kf_seq_show+0xa4/0xf0
  Sep 12 18:55:09 WBOX kernel:  seq_read_iter+0x123/0x450
  Sep 12 18:55:09 WBOX kernel:  new_sync_read+0x134/0x1c0
  Sep 12 18:55:09 WBOX kernel:  vfs_read+0x148/0x190
  Sep 12 18:55:09 WBOX kernel:  ksys_read+0x6f/0xf0
  Sep 12 18:55:09 WBOX kernel:  do_syscall_64+0x5c/0x90
  Sep 12 18:55:09 WBOX kernel:  ? exit_to_user_mode_prepare+0x16f/0x1d0
  Sep 12 18:55:09 WBOX kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
  Sep 12 18:55:09 WBOX kernel:  ? do_syscall_64+0x6b/0x90
  Sep 12 18:55:09 WBOX kernel:  ? do_syscall_64+0x6b/0x90
  Sep 12 18:55:09 WBOX kernel:  entry_SYSCALL_64_after_hwframe+0x63/0xcd
  Sep 12 18:55:09 WBOX kernel: RIP: 0033:0x7fde45207e2c
  Sep 12 18:55:09 WBOX kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 bd f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 >
  Sep 12 18:55:09 WBOX kernel: RSP: 002b:00007fff996056a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
  Sep 12 18:55:09 WBOX kernel: RAX: ffffffffffffffda RBX: 000055d3b59fd5c0 RCX: 00007fde45207e2c
  Sep 12 18:55:09 WBOX kernel: RDX: 0000000000001000 RSI: 000055d3b5a3d6e0 RDI: 000000000000000e
  Sep 12 18:55:09 WBOX kernel: RBP: 00007fff996057d0 R08: 0000000000000000 R09: 00007fde452ea240
  Sep 12 18:55:09 WBOX kernel: R10: 00007fde452ea240 R11: 0000000000000246 R12: 0000000000000000
  Sep 12 18:55:09 WBOX kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 000000000000000e
  Sep 12 18:55:09 WBOX kernel:  </TASK>
  Sep 12 18:55:09 WBOX kernel: Modules linked in: intel_tcc_cooling x86_pkg_temp_thermal intel_powerclamp xt_MASQUERADE coretemp nf_conntrack_netlink nfnetlink kvm_intel iptable>
  Sep 12 18:55:09 WBOX kernel:  snd_hda_core iwlmei snd_rawmidi intel_uncore snd_seq_device i2c_i801 mc snd_hwdep pcspkr psmouse e1000e i2c_smbus drm_buddy processor_thermal_dev>
  Sep 12 18:55:09 WBOX kernel:  mbcache jbd2 uas usb_storage hid_logitech_dj usbhid dm_mod serio_raw atkbd nvme libps2 vivaldi_fmap nvme_core crc32c_intel xhci_pci xhci_pci_rene>
  Sep 12 18:55:09 WBOX kernel: Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc>
  Sep 12 18:55:09 WBOX kernel: CR2: 0000000000000000
  Sep 12 18:55:09 WBOX kernel: ---[ end trace 0000000000000000 ]---
  Sep 12 18:55:09 WBOX kernel: RIP: 0010:0x0
  Sep 12 18:55:09 WBOX kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
  Sep 12 18:55:09 WBOX kernel: RSP: 0018:ffffa7fc40f5bcb8 EFLAGS: 00010202
  Sep 12 18:55:09 WBOX kernel: RAX: 0000000000000000 RBX: ffffffff97b84d08 RCX: ffffffff97b84d08
  Sep 12 18:55:09 WBOX kernel: RDX: ffffa7fc40f5bcc0 RSI: 0000000000000004 RDI: ffff8b7c04622800
  Sep 12 18:55:09 WBOX kernel: RBP: 0000000000000004 R08: ffff8b7c04622838 R09: ffff8b7c0e4dd000
  Sep 12 18:55:09 WBOX kernel: R10: ffffa7fc40f5be20 R11: 0000000000000001 R12: ffff8b7c06519000
  Sep 12 18:55:09 WBOX kernel: R13: ffff8b7c04622838 R14: ffffffff97b84d08 R15: ffff8b7c04622800
  Sep 12 18:55:09 WBOX kernel: FS:  00007fde44fbd3c0(0000) GS:ffff8b7f6f480000(0000) knlGS:0000000000000000
  Sep 12 18:55:09 WBOX kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  Sep 12 18:55:09 WBOX kernel: CR2: ffffffffffffffd6 CR3: 0000000100b2e001 CR4: 00000000003706e0
  Sep 12 18:55:09 WBOX org.gnome.Shell.desktop[940]: Window manager warning: Failed to parse saved session file: Failed to open file “/var/lib/gdm/.config/mutter/sessions/10dd14>
  Sep 12 18:55:09 WBOX gnome-shell[940]: Failed to launch ibus-daemon: Failed to execute child process “ibus-daemon” (No such file or directory)

I had no time to dig into this deeper, just downgraded to 5.17.9 version from Archlinux Archive.

Hope it will help.

loqs

Member

Registered: 2014-03-06

Posts: 18,045

Re: thinkpad crashes while idle - power management issue

@boxuser007 please try linux 5.19.8.arch1-1