vmwgfx crash after Linux update to 6.10.4-arch2-1

rdkehn · 2024-08-12 17:28:30

Hi Everyone,

I'm running Arch Linux as a guest in VMWare Workstation 17.5.2 build-23775571. I updated to Linux 6.10.4.-arch2-1 (from 6.10.2) today and now vmwgfx crashes on boot. I do not get the login prompt. Arch is running because I can SSH to it. My configuration boots to the console; I then run startxfce4 so I don't believe X is the cause. The dmesg output provided below. Any/All suggestions to resolve this problem will be appreciated.

Thanks,
...doug

[ 13.297084] ------------[ cut here ]------------
[ 13.297086] Command buffer error.
[ 13.297139] WARNING: CPU: 0 PID: 186 at drivers/gpu/drm/vmwgfx/vmwgfx_cmdbuf.c:399 vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx]
[ 13.297160] Modules linked in: uas usb_storage hid_generic usbhid mptspi sr_mod cdrom scsi_transport_spi vmwgfx serio_raw mptscsih ata_generic atkbd drm_ttm_helper libps2 pata_acpi vivaldi_fmap ttm mptbase crc32c_intel xhci_pci intel_agp xhci_pci_renesas ata_piix intel_gtt i8042 serio
[ 13.297172] CPU: 0 PID: 186 Comm: irq/16-vmwgfx Not tainted 6.10.4-arch2-1 #1 517ed45cc9c4492ee5d5bfc2d2fe6ef1f2e7a8eb
[ 13.297174] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 13.297175] RIP: 0010:vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx]
[ 13.297186] Code: 01 00 01 e8 ba 8c 4f f9 0f 0b 4c 89 ff e8 40 fb ff ff e9 9d fe ff ff 48 c7 c7 99 d9 3f c0 c6 05 52 2f 01 00 01 e8 98 8c 4f f9 <0f> 0b e9 1f fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90
[ 13.297187] RSP: 0018:ffffb9c1805e3d78 EFLAGS: 00010282
[ 13.297188] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000003
[ 13.297189] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[ 13.297190] RBP: ffff907fc8274c98 R08: 0000000000000000 R09: ffffb9c1805e3bf8
[ 13.297191] R10: ffff9086dbdfffa8 R11: 0000000000000003 R12: ffff907fc4db5b00
[ 13.297192] R13: ffff907fc83fd318 R14: ffff907fc8274c88 R15: ffff907fc83fd300
[ 13.297193] FS: 0000000000000000(0000) GS:ffff9086dbe00000(0000) knlGS:0000000000000000
[ 13.297194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 13.297194] CR2: 0000774dc57671ca CR3: 00000006b9e20005 CR4: 00000000003706f0
[ 13.297196] Call Trace:
[ 13.297198] <TASK>
[ 13.297199] ? vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx a4fe13044bca4eda782d964fb8c4ca15afb325e9]
[ 13.297209] ? __warn.cold+0x8e/0xe8
[ 13.297211] ? vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx a4fe13044bca4eda782d964fb8c4ca15afb325e9]
[ 13.297221] ? report_bug+0xff/0x140
[ 13.297222] ? console_unlock+0x84/0x130
[ 13.297225] ? handle_bug+0x3c/0x80
[ 13.297226] ? exc_invalid_op+0x17/0x70
[ 13.297227] ? asm_exc_invalid_op+0x1a/0x20
[ 13.297230] ? vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx a4fe13044bca4eda782d964fb8c4ca15afb325e9]
[ 13.297238] ? vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx a4fe13044bca4eda782d964fb8c4ca15afb325e9]
[ 13.297245] vmw_cmdbuf_man_process+0x5d/0x100 [vmwgfx a4fe13044bca4eda782d964fb8c4ca15afb325e9]
[ 13.297253] vmw_cmdbuf_irqthread+0x25/0x30 [vmwgfx a4fe13044bca4eda782d964fb8c4ca15afb325e9]
[ 13.297261] vmw_thread_fn+0x3a/0x70 [vmwgfx a4fe13044bca4eda782d964fb8c4ca15afb325e9]
[ 13.297271] irq_thread_fn+0x20/0x60
[ 13.297273] irq_thread+0x18a/0x270
[ 13.297274] ? __pfx_irq_thread_fn+0x10/0x10
[ 13.297276] ? __pfx_irq_thread_dtor+0x10/0x10
[ 13.297277] ? __pfx_irq_thread+0x10/0x10
[ 13.297278] kthread+0xcf/0x100
[ 13.297281] ? __pfx_kthread+0x10/0x10
[ 13.297282] ret_from_fork+0x31/0x50
[ 13.297285] ? __pfx_kthread+0x10/0x10
[ 13.297286] ret_from_fork_asm+0x1a/0x30
[ 13.297288] </TASK>
[ 13.297289] ---[ end trace 0000000000000000 ]---

gromit · 2024-08-12 17:38:19

This very much looks like kernel regression, which should be bisected and reported upstream to the kernel developers and the regression mailing list ..

Are you confident to do the bisection on your own or do you need some help?
If you want we could also provide you with pre-built kernel images for you to test

Good info to get you started is:
- https://docs.kernel.org/admin-guide/rep … sions.html
- https://wiki.archlinux.org/title/Kernel … egressions

rdkehn · 2024-08-13 13:40:33

Hi gromit,

Thanks for the links. I will gladly accept any additional advice, examples, etc. for the bisection.

gromit · 2024-08-13 14:28:29

Please try the following kernel:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.2.r467.gabd9c88-1-x86_64.pkg.tar.zst

pieska · 2024-08-13 20:07:09

kernel 6.10.3-arch1-2 works too, issue started with 6.10.4.arch2-1

gromit · 2024-08-13 20:19:28

Could you try the kernel above @pieska? It's the first bisection step out of just a few that we would need to test in order to find the culprit commit

Edit: I also had another look and the commit range in question contains multiple changes to the vmwgfx things so doing the bisection seems actually worth it:

$ git log --grep='vmwgfx' --pretty=fixes v6.10.3..v6.10.4
af6441e6f3d4 ("drm/vmwgfx: Fix handling of dumb buffers")
c98ab18b9f31 ("drm/vmwgfx: Fix a deadlock in dma buf fence polling")
20dc9c50e7af ("drm/vmwgfx: Trigger a modeset when the screen moves")
865cc600e1bb ("drm/vmwgfx: Fix overlay when using Screen Targets")
9b35b1ea2602 ("drm/vmwgfx: Make sure the screen surface is ref counted")

Last edited by gromit (2024-08-13 20:25:30)

pieska · 2024-08-14 15:50:46

sure, kernel boots and vmwgfx works:

Linux bisect 6.10.2-1-00467-gabd9c8843cbc #1 SMP PREEMPT_DYNAMIC Tue, 13 Aug 2024 14:09:12 +0000 x86_64 GNU/Linux

[root@bisect ~]# dmesg | grep vmwgfx
[ 2.818623] vmwgfx 0000:00:0f.0: vgaarb: deactivate vga console
[ 2.819183] vmwgfx 0000:00:0f.0: [drm] FIFO at 0x00000000ff000000 size is 8192 KiB
[ 2.819191] vmwgfx 0000:00:0f.0: [drm] VRAM at 0x00000000f0000000 size is 131072 KiB
[ 2.819202] vmwgfx 0000:00:0f.0: [drm] Running on SVGA version 2.
[ 2.819208] vmwgfx 0000:00:0f.0: [drm] Capabilities: rect copy, cursor, cursor bypass, cursor bypass 2, 8bit emulation, alpha cursor, extended fifo, multimon, pitchlock, irq mask, display topology, gmr, traces, gmr2, screen object 2, command buffers, command buffers 2, gbobject, dx, hp cmd queue, no bb restriction, cap2 register,
[ 2.819214] vmwgfx 0000:00:0f.0: [drm] Capabilities2: grow otable, intra surface copy, dx2, gb memsize 2, screendma reg, otable ptdepth2, non ms to ms stretchblt, cursor mob, mshint, cb max size 4mb, dx3, frame type, trace full fb, extra regs, lo staging,
[ 2.819352] vmwgfx 0000:00:0f.0: [drm] DMA map mode: Caching DMA mappings.
[ 2.819441] vmwgfx 0000:00:0f.0: [drm] Legacy memory limits: VRAM = 4096 KiB, FIFO = 256 KiB, surface = 0 KiB
[ 2.819442] vmwgfx 0000:00:0f.0: [drm] MOB limits: max mob size = 16384 KiB, max mob pages = 4096
[ 2.819444] vmwgfx 0000:00:0f.0: [drm] Max GMR ids is 64
[ 2.819445] vmwgfx 0000:00:0f.0: [drm] Max number of GMR pages is 65536
[ 2.819446] vmwgfx 0000:00:0f.0: [drm] Maximum display memory size is 16384 KiB
[ 2.823346] vmwgfx 0000:00:0f.0: [drm] Screen Target display unit initialized
[ 2.823909] vmwgfx 0000:00:0f.0: [drm] Fifo max 0x00040000 min 0x00001000 cap 0x0000077f
[ 2.827965] vmwgfx 0000:00:0f.0: [drm] Using command buffers with DMA pool.
[ 2.827970] vmwgfx 0000:00:0f.0: [drm] Available shader model: Legacy.
[ 2.837080] [drm] Initialized vmwgfx 2.20.0 20211206 for 0000:00:0f.0 on minor 0
[ 2.838212] fbcon: vmwgfxdrmfb (fb0) is primary device
[ 2.839092] vmwgfx 0000:00:0f.0: [drm] fb0: vmwgfxdrmfb frame buffer device

gromit · 2024-08-14 16:45:54

This is the next image (of approximately 8 more) to test:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.2.r700.g9bf1a4e-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 17:41:13

still good:

[root@bisect ~]# uname -a
Linux bisect 6.10.2-1-00700-g9bf1a4e3d4be #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 15:56:58 +0000 x86_64 GNU/Linux
[root@bisect ~]# dmesg | grep vmwgfx
[ 2.855229] vmwgfx 0000:00:0f.0: vgaarb: deactivate vga console
[ 2.855812] vmwgfx 0000:00:0f.0: [drm] FIFO at 0x00000000ff000000 size is 8192 KiB
[ 2.855821] vmwgfx 0000:00:0f.0: [drm] VRAM at 0x00000000f0000000 size is 131072 KiB
[ 2.855832] vmwgfx 0000:00:0f.0: [drm] Running on SVGA version 2.
[ 2.855838] vmwgfx 0000:00:0f.0: [drm] Capabilities: rect copy, cursor, cursor bypass, cursor bypass 2, 8bit emulation, alpha cursor, extended fifo, multimon, pitchlock, irq mask, display topology, gmr, traces, gmr2, screen object 2, command buffers, command buffers 2, gbobject, dx, hp cmd queue, no bb restriction, cap2 register,
[ 2.855843] vmwgfx 0000:00:0f.0: [drm] Capabilities2: grow otable, intra surface copy, dx2, gb memsize 2, screendma reg, otable ptdepth2, non ms to ms stretchblt, cursor mob, mshint, cb max size 4mb, dx3, frame type, trace full fb, extra regs, lo staging,
[ 2.855992] vmwgfx 0000:00:0f.0: [drm] DMA map mode: Caching DMA mappings.
[ 2.856074] vmwgfx 0000:00:0f.0: [drm] Legacy memory limits: VRAM = 4096 KiB, FIFO = 256 KiB, surface = 0 KiB
[ 2.856076] vmwgfx 0000:00:0f.0: [drm] MOB limits: max mob size = 16384 KiB, max mob pages = 4096
[ 2.856077] vmwgfx 0000:00:0f.0: [drm] Max GMR ids is 64
[ 2.856078] vmwgfx 0000:00:0f.0: [drm] Max number of GMR pages is 65536
[ 2.856079] vmwgfx 0000:00:0f.0: [drm] Maximum display memory size is 16384 KiB
[ 2.860006] vmwgfx 0000:00:0f.0: [drm] Screen Target display unit initialized
[ 2.860517] vmwgfx 0000:00:0f.0: [drm] Fifo max 0x00040000 min 0x00001000 cap 0x0000077f
[ 2.863440] vmwgfx 0000:00:0f.0: [drm] Using command buffers with DMA pool.
[ 2.863446] vmwgfx 0000:00:0f.0: [drm] Available shader model: Legacy.
[ 2.874109] [drm] Initialized vmwgfx 2.20.0 20211206 for 0000:00:0f.0 on minor 0
[ 2.875141] fbcon: vmwgfxdrmfb (fb0) is primary device
[ 2.877960] vmwgfx 0000:00:0f.0: [drm] fb0: vmwgfxdrmfb frame buffer device

gromit · 2024-08-14 18:29:27

Please try the following image:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r7.g49dba57-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 18:42:41

Linux bisect 6.10.3-1-00007-g49dba573ef7c #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 17:50:21 +0000 x86_64 GNU/Linux

works

gromit · 2024-08-14 19:01:04

Please test the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r65.g6eac2b6-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 19:04:32

Linux bisect 6.10.3-1-00065-g6eac2b62975f #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 18:47:27 +0000 x86_64 GNU/Linux

works

gromit · 2024-08-14 19:18:52

Nice, if we keep this rate we will finish the bisection today \o/

Please try the following kernel:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r94.g2261285-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 19:24:15

I automated things, the script just waits for your link to kick in... I had to stopped it now because:

Linux bisect 6.10.3-1-00094-g226128518a84 #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 19:07:00 +0000 x86_64 GNU/Linux

fails with

[ 2.832305] ------------[ cut here ]------------
[ 2.832306] Command buffer error.
[ 2.832318] WARNING: CPU: 0 PID: 344 at drivers/gpu/drm/vmwgfx/vmwgfx_cmdbuf.c:399 vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx]

gromit · 2024-08-14 19:38:42

Please test the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r79.g6eb1d25-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 19:42:18

Linux bisect 6.10.3-1-00079-g6eb1d257986f #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 19:27:58 +0000 x86_64 GNU/Linux

works

gromit · 2024-08-14 19:57:52

Please test:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r86.gece30d0-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 20:00:22

Linux bisect 6.10.3-1-00086-gece30d03d281 #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 19:44:59 +0000 x86_64 GNU/Linux

works

gromit · 2024-08-14 20:14:36

Please test:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r90.g619186e-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 20:17:15

Linux bisect 6.10.3-1-00090-g619186e7964e #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 20:01:19 +0000 x86_64 GNU/Linux

works

gromit · 2024-08-14 20:29:53

Please try:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r92.gaf6441e-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 20:33:30

Linux bisect 6.10.3-1-00092-gaf6441e6f3d4 #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 20:18:13 +0000 x86_64 GNU/Linux

fails with:

[ 2.839905] Command buffer error.
[ 2.839915] WARNING: CPU: 0 PID: 343 at drivers/gpu/drm/vmwgfx/vmwgfx_cmdbuf.c:399 vmw_cmdbuf_ctx_process+0x268/0x270 [vmwgfx]

gromit · 2024-08-14 20:49:49

Please test the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-v6.10.3.r91.gc98ab18-1-x86_64.pkg.tar.zst

pieska · 2024-08-14 20:54:52

Linux bisect 6.10.3-1-00091-gc98ab18b9f31 #1 SMP PREEMPT_DYNAMIC Wed, 14 Aug 2024 20:35:28 +0000 x86_64 GNU/Linux

works

Arch Linux

#1 2024-08-12 17:28:30

vmwgfx crash after Linux update to 6.10.4-arch2-1

#2 2024-08-12 17:38:19

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#3 2024-08-13 13:40:33

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#4 2024-08-13 14:28:29

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#5 2024-08-13 20:07:09

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#6 2024-08-13 20:19:28

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#7 2024-08-14 15:50:46

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#8 2024-08-14 16:45:54

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#9 2024-08-14 17:41:13

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#10 2024-08-14 18:29:27

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#11 2024-08-14 18:42:41

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#12 2024-08-14 19:01:04

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#13 2024-08-14 19:04:32

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#14 2024-08-14 19:18:52

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#15 2024-08-14 19:24:15

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#16 2024-08-14 19:38:42

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#17 2024-08-14 19:42:18

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#18 2024-08-14 19:57:52

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#19 2024-08-14 20:00:22

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#20 2024-08-14 20:14:36

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#21 2024-08-14 20:17:15

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#22 2024-08-14 20:29:53

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#23 2024-08-14 20:33:30

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#24 2024-08-14 20:49:49

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

#25 2024-08-14 20:54:52

Re: vmwgfx crash after Linux update to 6.10.4-arch2-1

Board footer