You are not logged in.
- Topics: Active | Unanswered
#1 2019-01-30 13:19:58
- Shara
- Member
- Registered: 2012-09-01
- Posts: 18
100% of one core used by system after last update. Help with diagnost
After last full update i have lost wifi in NetworkManager.
I boot fallback-initramfs, wifi works. Then I made "mkinitcpio -p linux" and now wifi is working.
Then I found that CPU always loaded.
CPU usage in nmon looks like this:
https://i.imgur.com/hIh5dM3.png
Core 3 always loaded near 90%. I tried to find which process use 3 core, but I can't. There is nothing in top. There is nothing in oter system monitor tools.
Sometimes and after boot nmon CPU usage looks like this:
https://i.imgur.com/E2TQdas.png
But after some time, core load by %Sys stuck at 90% again.
How to dig this problem?
Mod note: Replaced oversized images with URLs. Please read the Code of Conduct and only post thumbnails. -- WorMzy
Last edited by WorMzy (2019-01-30 14:05:14)
Offline
#2 2019-01-30 13:59:15
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,052
Re: 100% of one core used by system after last update. Help with diagnost
Please replace the oversized images w/ links or the actual text (in code tags) - it's some shell output after all.
The lionshare of the core is taken by "sys" what I assume nmons representation for the kernel is, so the CPU is required by the kernel or a kernel module.
Because of the described precondition, what's the output of:
uname -a
pacman -S linux
lsmod # cause it's most likely one of the modulesAlso, is this w/ the fallback initramfs or the lean one?
Offline
#3 2019-01-31 17:43:38
- Shara
- Member
- Registered: 2012-09-01
- Posts: 18
Re: 100% of one core used by system after last update. Help with diagnost
seth, I can't find how to manually repeat problem.
uname:
Linux len 4.20.5-arch1-1-ARCH #1 SMP PREEMPT Sat Jan 26 12:59:18 UTC 2019 x86_64 GNU/Linux
lsmod:
Module Size Used by
ccm 20480 3
rfcomm 86016 4
fuse 131072 3
bnep 28672 2
bbswitch 20480 0
uvcvideo 114688 0
videobuf2_vmalloc 20480 1 uvcvideo
videobuf2_memops 20480 1 videobuf2_vmalloc
snd_hda_codec_hdmi 61440 1
videobuf2_v4l2 28672 1 uvcvideo
btusb 57344 0
videobuf2_common 57344 2 videobuf2_v4l2,uvcvideo
snd_hda_codec_realtek 122880 1
btrtl 20480 1 btusb
videodev 229376 3 videobuf2_v4l2,uvcvideo,videobuf2_common
btbcm 16384 1 btusb
snd_hda_codec_generic 90112 1 snd_hda_codec_realtek
btintel 28672 1 btusb
bluetooth 651264 31 btrtl,btintel,btbcm,bnep,btusb,rfcomm
media 57344 4 videodev,videobuf2_v4l2,uvcvideo,videobuf2_common
i915 2109440 7
intel_rapl 28672 0
ecdh_generic 24576 1 bluetooth
wl 6463488 0
x86_pkg_temp_thermal 20480 0
intel_powerclamp 20480 0
kvmgt 32768 0
coretemp 20480 0
vfio_mdev 16384 0
mdev 24576 2 kvmgt,vfio_mdev
vfio_iommu_type1 32768 0
vfio 36864 3 kvmgt,vfio_mdev,vfio_iommu_type1
kvm_intel 245760 0
i2c_algo_bit 16384 1 i915
drm_kms_helper 208896 1 i915
drm 499712 8 drm_kms_helper,i915
arc4 16384 2
kvm 741376 2 kvmgt,kvm_intel
rtl8723ae 131072 0
intel_gtt 24576 1 i915
irqbypass 16384 1 kvm
agpgart 53248 2 intel_gtt,drm
btcoexist 221184 1 rtl8723ae
crct10dif_pclmul 16384 1
rtl8723_common 28672 1 rtl8723ae
joydev 28672 0
crc32_pclmul 16384 0
syscopyarea 16384 1 drm_kms_helper
mousedev 24576 0
sysfillrect 16384 1 drm_kms_helper
rtl_pci 36864 1 rtl8723ae
ghash_clmulni_intel 16384 0
snd_hda_intel 49152 3
sysimgblt 16384 1 drm_kms_helper
snd_hda_codec 155648 4 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec_realtek
pktcdvd 49152 1
fb_sys_fops 16384 1 drm_kms_helper
rtlwifi 126976 4 rtl_pci,btcoexist,rtl8723ae,rtl8723_common
snd_hda_core 98304 5 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek
cryptd 28672 1 ghash_clmulni_intel
snd_hwdep 16384 1 snd_hda_codec
intel_cstate 16384 0
snd_pcm 135168 4 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hda_core
intel_uncore 135168 0
mac80211 946176 2 rtl_pci,rtlwifi
snd_timer 40960 1 snd_pcm
input_leds 16384 0
intel_rapl_perf 16384 0
snd 102400 14 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek,snd_timer,snd_pcm
mac_hid 16384 0
iTCO_wdt 16384 0
mei_me 45056 0
cfg80211 778240 3 wl,rtlwifi,mac80211
iTCO_vendor_support 16384 1 iTCO_wdt
soundcore 16384 1 snd
mei 114688 1 mei_me
psmouse 172032 0
wmi_bmof 16384 0
rtsx_pci_ms 24576 0
rfkill 28672 5 bluetooth,cfg80211
wmi 32768 1 wmi_bmof
memstick 20480 1 rtsx_pci_ms
alx 53248 0
lpc_ich 28672 0
i2c_i801 36864 0
evdev 20480 14
pcc_cpufreq 20480 0
mdio 16384 1 alx
battery 24576 0
ac 16384 0
vboxnetflt 32768 0
vboxnetadp 28672 0
vboxpci 28672 0
vboxdrv 499712 3 vboxpci,vboxnetadp,vboxnetflt
sg 40960 0
crypto_user 20480 0
ip_tables 32768 0
x_tables 49152 1 ip_tables
ext4 749568 2
crc32c_generic 16384 0
crc16 16384 2 bluetooth,ext4
mbcache 16384 1 ext4
jbd2 131072 1 ext4
fscrypto 32768 1 ext4
sr_mod 28672 1
cdrom 73728 2 pktcdvd,sr_mod
sd_mod 57344 4
ata_generic 16384 0
pata_acpi 16384 0
rtsx_pci_sdmmc 32768 0
mmc_core 180224 1 rtsx_pci_sdmmc
serio_raw 20480 0
atkbd 36864 0
libps2 20480 2 atkbd,psmouse
ata_piix 36864 4
libata 278528 3 ata_piix,pata_acpi,ata_generic
crc32c_intel 24576 2
scsi_mod 258048 4 sd_mod,libata,sg,sr_mod
ehci_pci 20480 0
rtsx_pci 77824 2 rtsx_pci_sdmmc,rtsx_pci_ms
ehci_hcd 98304 1 ehci_pci
i8042 32768 0
serio 28672 6 serio_raw,atkbd,psmouse,i8042How to find which module make cpu load?
Last edited by Shara (2019-01-31 17:43:59)
Offline
#4 2019-01-31 20:47:47
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,052
Re: 100% of one core used by system after last update. Help with diagnost
Unless you can reproduce the problem - not at all.
vbox* is a constant contender for trouble causes.
Offline
#5 2019-02-14 16:36:12
- Shara
- Member
- Registered: 2012-09-01
- Posts: 18
Re: 100% of one core used by system after last update. Help with diagnost
Happens again. Same problem - near 100% of core 3. Can i check which kernel model uses it?
Maybe this log will help?
journalctl -b -e
kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0]
kernel: Modules linked in: hid_logitech ff_memless hid_generic usbhid hid ccm rfcomm fuse bnep bbswitch(OE) snd_hda_codec_hdmi uvcvideo snd_hda_codec_realtek snd_hda_codec_>
kernel: vboxnetflt(OE) vboxnetadp(OE) vboxpci(OE) vboxdrv(OE) sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sr_mod cdrom sd_mod ata_gen>
kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P OE 4.20.5-arch1-1-ARCH #1
kernel: Hardware name: Quanta TWC/TWC, BIOS DR171 04/01/2013
kernel: RIP: 0010:_raw_spin_unlock_irqrestore+0x11/0x40
kernel: Code: 8b 05 43 31 1d 60 85 c0 74 01 c3 e8 66 0b 7c ff c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 bf 01 00 00 00 >
kernel: RSP: 0018:ffff98f7db0834e8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13
kernel: RAX: 00000000000000ff RBX: ffff98f69b03aa00 RCX: ffff98f7d97915c8
kernel: RDX: 0000000000000001 RSI: 0000000000000216 RDI: 0000000000000216
kernel: RBP: ffff98f7d9790780 R08: ffff98f7db08352f R09: ffffffffc0c46fb0
kernel: R10: 0000000000000088 R11: 0000000000000000 R12: ffff98f791eb4cd2
kernel: R13: 0000000000000001 R14: ffff98f6b70cbcc0 R15: ffff98f7d9791580
kernel: FS: 0000000000000000(0000) GS:ffff98f7db080000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007fcccfa76000 CR3: 000000006c80a003 CR4: 00000000001606e0
kernel: Call Trace:
kernel: <IRQ>
kernel: rtl_pci_tx+0x25d/0x480 [rtl_pci]
kernel: rtl_op_tx+0xad/0xd0 [rtlwifi]
kernel: ieee80211_tx_frags+0x14c/0x220 [mac80211]
kernel: __ieee80211_subif_start_xmit+0xaaa/0xb80 [mac80211]
kernel: ? _raw_spin_lock_bh+0x28/0x30
kernel: ieee80211_subif_start_xmit+0x43/0x2e0 [mac80211]
kernel: ? skb_checksum+0x32/0x50
kernel: ? csum_block_add_ext+0x20/0x20
kernel: ? reqsk_fastopen_remove+0x150/0x150
kernel: dev_hard_start_xmit+0xa5/0x240
kernel: sch_direct_xmit+0x150/0x340
kernel: __dev_queue_xmit+0x2f6/0x910
kernel: ? __cgroup_bpf_run_filter_skb+0x16b/0x240
kernel: ip_finish_output2+0x29b/0x420
kernel: ? ip_finish_output+0x100/0x250
kernel: ip_output+0x73/0x110
kernel: __ip_queue_xmit+0x162/0x420
kernel: __tcp_transmit_skb+0x59d/0xa90
kernel: tcp_rcv_established+0x53d/0x630
kernel: tcp_v4_do_rcv+0x131/0x1d0
kernel: tcp_v4_rcv+0xc33/0xdc0
kernel: ip_local_deliver_finish+0xa1/0x210
kernel: ip_local_deliver+0x59/0x120
kernel: ? ip_rcv_finish_core.isra.0+0x18c/0x3d0
kernel: ip_rcv+0x51/0x100
kernel: __netif_receive_skb_one_core+0x52/0x70
kernel: netif_receive_skb_internal+0x4a/0x140
kernel: ? ieee80211_data_to_8023_exthdr+0x36d/0x3c0 [cfg80211]
kernel: ieee80211_deliver_skb+0x205/0x250 [mac80211]
kernel: ieee80211_rx_handlers+0xba2/0x28f0 [mac80211]
kernel: ? _raw_spin_lock_irqsave+0x25/0x50
kernel: ? lock_timer_base+0x67/0x80
kernel: ieee80211_prepare_and_rx_handle+0xb98/0x1260 [mac80211]
kernel: ieee80211_rx_napi+0x38e/0xa40 [mac80211]
kernel: ? _raw_spin_unlock_irqrestore+0x11/0x40
kernel: ieee80211_tasklet_handler+0xbd/0xd0 [mac80211]
kernel: tasklet_action_common.isra.2+0x49/0xb0
kernel: __do_softirq+0x112/0x356
kernel: irq_exit+0xd9/0xf0
kernel: do_IRQ+0x9d/0x100
kernel: common_interrupt+0xf/0xf
kernel: </IRQ>
kernel: RIP: 0010:cpuidle_enter_state+0xb9/0x360
kernel: Code: e8 ac 1f a3 ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 7e 02 00 00 31 ff e8 9e 2f a9 ff fb 66 0f 1f 44 00 00 <85> ed 0f 88 4c 02 00 00 48 b8 >
kernel: RSP: 0018:ffffb4e3806e7ea0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd
kernel: RAX: ffff98f7db080000 RBX: 0000068dc0924bb8 RCX: 000000000000001f
kernel: RDX: 0000068dc0924bb8 RSI: ffffffffa06a41e6 RDI: ffffffffa06ac3e0
kernel: RBP: 0000000000000005 R08: 0000000000000002 R09: 0000000000021480
kernel: R10: 0000105dbd524fbb R11: ffff98f7db0a0b44 R12: ffffffffa08b4e98
kernel: R13: ffff98f7db0aad00 R14: 0000000000000005 R15: 0000000000000000
kernel: do_idle+0x217/0x250
kernel: cpu_startup_entry+0x19/0x20
kernel: start_secondary+0x1aa/0x200
kernel: secondary_startup_64+0xa4/0xb0Last edited by Shara (2019-02-14 16:44:57)
Offline
#6 2019-02-14 17:06:47
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,052
Re: 100% of one core used by system after last update. Help with diagnost
Because of the initial post, I'd try to unload rtl8723ae and see whether that makes a difference.
You can profile the kernel w/ https://www.archlinux.org/packages/comm … 6_64/perf/
Edit, try loading the module w/
rtl8723ae.swenc=1 rtl8723ae.ips=0 rtl8723ae.swlps=0 rtl8723ae.fwlps=0 rtl8723ae.aspm=0(just a hunch from the kernel trace)
Last edited by seth (2019-02-14 17:13:56)
Offline
#7 2019-02-14 17:40:25
- loqs
- Member
- Registered: 2014-03-06
- Posts: 17,325
Re: 100% of one core used by system after last update. Help with diagnost
Offline
#8 2019-02-14 20:28:06
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,052
Re: 100% of one core used by system after last update. Help with diagnost
Isn't that one in userspace?
There is nothing in top. There is nothing in oter system monitor tools.
Sometimes and after boot nmon CPU usage looks like this:
https://i.imgur.com/E2TQdas.pngBut after some time, core load by %Sys stuck at 90% again.
Offline
#9 2019-02-26 20:37:42
- Shara
- Member
- Registered: 2012-09-01
- Posts: 18
Re: 100% of one core used by system after last update. Help with diagnost
Happens again. Yep, it's "rtl8723ae" module makes problems, but those arguments not solve them
sudo rmmod rtl8723ae
CPU3 load by SYS dropped to 0%
sudo modprobe -v rtl8723ae
CPU3 load by SYS 0.5% when WLAN off
CPU3 load by SYS 80% when WLAN on
sudo modprobe -v rtl8723ae swenc=1 ips=0 swlps=0 fwlps=0 aspm=0
CPU3 load by SYS 0.5% when WLAN off
CPU3 load by SYS 80% when WLAN onUPD. That's bug https://github.com/lwfinger/rtlwifi_new/issues/216
Last edited by Shara (2019-02-26 20:48:54)
Offline
#10 2019-02-26 20:47:02
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,052
Re: 100% of one core used by system after last update. Help with diagnost
Offline
#11 2019-02-26 20:50:23
- Shara
- Member
- Registered: 2012-09-01
- Posts: 18
Re: 100% of one core used by system after last update. Help with diagnost
Yep, found this. Sadness ¯\_(ツ)_/¯
Offline