Debugging daily, sporadic kernel panic

iceandsnow · 2024-05-06 12:29:43

Hi,

I get about a kernel panic with the latest kernel once every day. I now switched to LTS and I think it also happens there, but I still need to figure it out in detail.

I'm surprised, because I run standard hardware (Lenovo E14).

The panic only happens:
* when I'm not plugged in an AC outlet
* when I'm not actively typing or doing something

All I see is the rapidly blinking light on the shift key. How do I go about starting to figure out what causes the kernel panic?

Tx.

seth · 2024-05-06 16:41:50

I run standard hardware (Lenovo E14).

lspci -k
lsusb
lscpu

only happens:
* when I'm not plugged in an AC outlet
* when I'm not actively typing or doing something

Sounds power saving related, assuming it's not just the CPU you'd try to limit the c-states) do you use sth. like TLP or powertop in auto mode?

You're probably not gonna be able to reboto using the https://wiki.archlinux.org/title/Keyboa … el_(SysRq) ?
You'd then have to employ https://wiki.archlinux.org/title/Kdump do debug the kernel if nothing about your setup sticks out.

iceandsnow · 2024-05-06 17:40:16

Sounds power saving related, assuming it's not just the CPU you'd try to limit the c-states) do you use sth. like TLP or powertop in auto mode?

I think that what it is. I've never really used TLP and I'm now using it extensively. But even a minimal configuration that just limits the CPU in battery mode is hanging the system:

CPU_SCALING_GOVERNOR_ON_AC=performance
CPU_SCALING_GOVERNOR_ON_BAT=powersave

CPU_SCALING_MIN_FREQ_ON_AC=400000
CPU_SCALING_MAX_FREQ_ON_AC=4600000
CPU_SCALING_MIN_FREQ_ON_BAT=400000
CPU_SCALING_MAX_FREQ_ON_BAT=1200000

PLATFORM_PROFILE_ON_AC=performance
PLATFORM_PROFILE_ON_BAT=balanced

Here's the output of lspci, lscpu, lsusb:

## lspci -k

00:00.0 Host bridge: Intel Corporation Raptor Lake-P/U 2p+8e cores Host Bridge/DRAM Controller (rev 01)
	Subsystem: Lenovo Device 50d1
00:02.0 VGA compatible controller: Intel Corporation Raptor Lake-P [Iris Xe Graphics] (rev 04)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: i915
	Kernel modules: i915
00:04.0 Signal processing controller: Intel Corporation Raptor Lake Dynamic Platform and Thermal Framework Processor Participant (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: proc_thermal_pci
	Kernel modules: processor_thermal_device_pci
00:06.0 PCI bridge: Intel Corporation Raptor Lake PCIe 4.0 Graphics Port (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: pcieport
00:07.0 PCI bridge: Intel Corporation Device a73f (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: pcieport
00:08.0 System peripheral: Intel Corporation GNA Scoring Accelerator module (rev 01)
	Subsystem: Lenovo Device 50d1
00:0d.0 USB controller: Intel Corporation Raptor Lake-P Thunderbolt 4 USB Controller (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
00:0d.2 USB controller: Intel Corporation Raptor Lake-P Thunderbolt 4 NHI #0 (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: thunderbolt
	Kernel modules: thunderbolt
00:14.0 USB controller: Intel Corporation Alder Lake PCH USB 3.2 xHCI Host Controller (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
00:14.2 RAM memory: Intel Corporation Alder Lake PCH Shared SRAM (rev 01)
	Subsystem: Lenovo Device 50d1
00:14.3 Network controller: Intel Corporation Raptor Lake PCH CNVi WiFi (rev 01)
	Subsystem: Intel Corporation Device 0074
	Kernel driver in use: iwlwifi
	Kernel modules: iwlwifi
00:15.0 Serial bus controller: Intel Corporation Alder Lake PCH Serial IO I2C Controller #0 (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: intel-lpss
	Kernel modules: intel_lpss_pci
00:15.2 Serial bus controller: Intel Corporation Alder Lake PCH Serial IO I2C Controller #2 (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: intel-lpss
	Kernel modules: intel_lpss_pci
00:16.0 Communication controller: Intel Corporation Alder Lake PCH HECI Controller (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: mei_me
	Kernel modules: mei_me
00:1f.0 ISA bridge: Intel Corporation Raptor Lake LPC/eSPI Controller (rev 01)
	Subsystem: Lenovo Device 50d1
00:1f.3 Multimedia audio controller: Intel Corporation Raptor Lake-P/U/H cAVS (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: sof-audio-pci-intel-tgl
	Kernel modules: snd_hda_intel, snd_sof_pci_intel_tgl
00:1f.4 SMBus: Intel Corporation Alder Lake PCH-P SMBus Host Controller (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-P PCH SPI Controller (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: intel-spi
	Kernel modules: spi_intel_pci
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (23) I219-V (rev 01)
	Subsystem: Lenovo Device 50d1
	Kernel driver in use: e1000e
	Kernel modules: e1000e
01:00.0 Non-Volatile memory controller: Micron Technology Inc 2450 NVMe SSD [HendrixV] (DRAM-less) (rev 01)
	Subsystem: Micron Technology Inc Device 2100
	Kernel driver in use: nvme
	Kernel modules: nvme

## lsusb

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 30c9:00a7 Luxvisions Innotech Limited Integrated Camera
Bus 003 Device 003: ID 8087:0026 Intel Corp. AX201 Bluetooth
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub


## lscpu

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   12
  On-line CPU(s) list:    0-11
Vendor ID:                GenuineIntel
  Model name:             13th Gen Intel(R) Core(TM) i5-1335U
    CPU family:           6
    Model:                186
    Thread(s) per core:   2
    Core(s) per socket:   10
    Socket(s):            1
    Stepping:             3
    CPU(s) scaling MHz:   28%
    CPU max MHz:          4600.0000
    CPU min MHz:          400.0000
    BogoMIPS:             4993.00
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse ss
                          e2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopol
                          ogy nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse
                          3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
                           lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ep
                          t vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel
                          _pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts 
                          hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq r
                          dpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
Virtualization features:  
  Virtualization:         VT-x
Caches (sum of all):      
  L1d:                    352 KiB (10 instances)
  L1i:                    576 KiB (10 instances)
  L2:                     6.5 MiB (4 instances)
  L3:                     12 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-11
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Mitigation; Clear Register File
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_D
                          IS_S
  Srbds:                  Not affected
  Tsx async abort:        Not affected

seth · 2024-05-06 21:09:45

Could you confirm that TLP (and that config segment specifically) is the cause?
Do you have https://archlinux.org/packages/extra/x8 … es-daemon/ installed?
I don't assume it's CPU_SCALING_MAX_FREQ_ON_BAT and you didn't change the min - is the cause CPU_SCALING_GOVERNOR_ON_BAT ?

Arch Linux

#1 2024-05-06 12:29:43

Debugging daily, sporadic kernel panic

#2 2024-05-06 16:41:50

Re: Debugging daily, sporadic kernel panic

#3 2024-05-06 17:40:16

Re: Debugging daily, sporadic kernel panic

#4 2024-05-06 21:09:45

Re: Debugging daily, sporadic kernel panic

Board footer