Arch Linux

pinsl

Member

Registered: 2016-02-07

Posts: 7

[SOLVED] SMB kernel panic

Hi,

since a few weeks the kernel of my arch server panics after some data (~2GB) has been written to a samba share.

Here's the output:

[ 1713.000554] detected buffer overflow in strcat
[ 1713.001319] kernel BUG at lib/string.c:1052!
[ 1713.001902] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 1713.002588] Modules linked in: arc4 md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache cfg80211 rfkill 8021q mrp bochs_drm ttm drm_kms_helper hid_generic joydev mousedev drm usbhid hid input_leds intel_agp led_class syscopyarea sysfillrect psmouse intel_gtt sysimgblt fb_sys_fops pcspkr shpchp agpgart i2c_piix4 rtc_cmos evdev mac_hid qemu_fw_cfg ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto ata_generic pata_acpi virtio_net virtio_blk virtio_balloon serio_raw atkbd libps2 ata_piix floppy i8042 serio uhci_hcd ehci_pci ehci_hcd libata scsi_mod usbcore usb_common virtio_pci virtio_ring virtio
[ 1713.010401] CPU: 0 PID: 7172 Comm: deluged Not tainted 4.16.9-1-ARCH #1
[ 1713.011327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
[ 1713.013162] RIP: 0010:fortify_panic+0xf/0x1a
[ 1713.013876] RSP: 0018:ffff8f3f40ba7a88 EFLAGS: 00010246
[ 1713.014754] RAX: 0000000000000022 RBX: 0000000000000000 RCX: 0000000000000002
[ 1713.015695] RDX: 0000000000000000 RSI: ffffffffb9ed7a7a RDI: 00000000ffffffff
[ 1713.016559] RBP: ffff8b8e0a608f28 R08: 0000000000000000 R09: 000000000000022f
[ 1713.017576] R10: ffff8f3f4244f020 R11: 0000000000000001 R12: 0000000000000000
[ 1713.018452] R13: 0000000000000001 R14: ffff8b8e05891d80 R15: ffff8b8db5a5b400
[ 1713.019418] FS:  00007f38ba7f3700(0000) GS:ffff8b8e3fc00000(0000) knlGS:0000000000000000
[ 1713.020178] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1713.020931] CR2: 000055b259623650 CR3: 0000000127c1a000 CR4: 00000000000006f0
[ 1713.021889] Call Trace:
[ 1713.022248]  smb21_set_oplock_level.cold.10+0xc/0xc [cifs]
[ 1713.023025]  ? smb3_set_oplock_level+0x1d/0x80 [cifs]
[ 1713.023694]  ? smb2_set_fid+0x6b/0xa0 [cifs]
[ 1713.024269]  ? cifs_new_fileinfo+0x259/0x3a0 [cifs]
[ 1713.024926]  ? cifs_open+0x3d3/0x900 [cifs]
[ 1713.025437]  ? autofs4_d_manage+0xa5/0x1a0
[ 1713.026013]  ? generic_permission+0x10a/0x190
[ 1713.026662]  ? cifsFileInfo_get+0x30/0x30 [cifs]
[ 1713.027315]  ? do_dentry_open+0x14c/0x300
[ 1713.027864]  ? do_dentry_open+0x14c/0x300
[ 1713.028478]  ? path_openat+0x30c/0x1240
[ 1713.029002]  ? do_filp_open+0x93/0x100
[ 1713.029565]  ? __check_object_size+0xfb/0x180
[ 1713.030186]  ? do_sys_open+0x186/0x210
[ 1713.030869]  ? do_syscall_64+0x74/0x190
[ 1713.031398]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 1713.032247] Code: d0 48 89 cf 48 0f 42 e8 48 89 ea e8 56 6d 00 00 c6 04 28 00 48 89 d8 5b 5d c3 0f 0b 48 89 fe 48 c7 c7 50 a8 ee b9 e8 75 be 9c ff <0f> 0b 90 90 90 90 90 90 90 90 90 55 48 89 fa 48 89 fd 31 c9 53 
[ 1713.034912] RIP: fortify_panic+0xf/0x1a RSP: ffff8f3f40ba7a88
[ 1713.035786] ---[ end trace e413c31727ccfc02 ]---
[ 1713.036465] note: deluged[7172] exited with preempt_count 1

I have no idea how to debug this.
Any help would be appreciated.

Last edited by pinsl (2018-10-01 12:39:09)

loqs

Member

Registered: 2014-03-06

Posts: 17,322

Re: [SOLVED] SMB kernel panic

Did the issue start with the update of the kernel to the 4.16 series or on a later upgrade?   You can check by finding the first 4.16 kernel update
in pacman.log then downgrade the kernel to that release and see if you can recreate the issue.
Edit:
Oh and welcome to the arch linux forums pinsl.

Last edited by loqs (2018-05-22 11:47:55)

pinsl

Member

Registered: 2016-02-07

Posts: 7

Re: [SOLVED] SMB kernel panic

4.15.15-1 and 4.16.11-1 also panic

I normally run the LTS kernel so an upgrade of linux-lts introduced the issue.
I just ran the normal kernel for the last few weeks so I'd get the fix faster.

After trying out some kernels I found the first kernel that showed the issue:

6920:[2018-02-23 23:15] [ALPM] upgraded linux-lts (4.9.76-1 -> 4.14.20-1)

4.9.76-1-lts does not panic
4.14.20-1-lts panics

4.14.20-1-lts also introduced this error message which is not present on 4.16 though:

[    6.301837] CIFS VFS: ioctl error in smb2_get_dfs_refer rc=-2

Last edited by pinsl (2018-05-26 11:37:37)

seth

Member

Registered: 2012-09-03

Posts: 51,029

Re: [SOLVED] SMB kernel panic

Are you sure 4.9.76 isn't just a SMBv1 connection?

Can you please elaborate on the topology?
What exactly is SMB server and client(s) here (OS, implementation - ie. cifs or libsmb - config, esp. protocol version restrictions) and where do you count the (constant?) amount of data to failure? (client traffic, server traffic; many files or one)
Are there error messages on the other system (client if arch is server et vv.)?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

pinsl

Member

Registered: 2016-02-07

Posts: 7

Re: [SOLVED] SMB kernel panic

I have Arch and a Debian NAS (openmediavault) running as VMs on the same machine.

This is the fstab entry of the samba share:

//nas/volume	/mnt/volume	cifs	uid=admin,gid=users,file_mode=0774,dir_mode=0774,user,noauto,x-systemd.automount,credentials=/etc/naspw,vers=3.0	0 0

So I guess I am forcing it to use version 3.

To reproduce the kernel panic i just let torrents download and wait.
Only a few big files are being written.

I don't see any error message on the server.

seth

Member

Registered: 2012-09-03

Posts: 51,029

Re: [SOLVED] SMB kernel panic

Ok, so the bottom line is a fortify panic during network load on virtual machines and it does not seem related to the SMB version.

I'd try
- using SMB v2.0
- load the cifs module w/ oplocks disabled
to rule out a direct code issue.

If this doesn't "fix" it, but you simply get different backtraces, you could try playing w/ the buffers (see "modinfo cifs").
You should also try the behavior on real HW (to rule out bad interaction between cifs and virtio)
Also check meminfo during the network load (whether you're running out of RAM or some item like eg. slab or kernel stack raises abnormally.

What's btw. the host system?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

pinsl

Member

Registered: 2016-02-07

Posts: 7

Re: [SOLVED] SMB kernel panic

I didn't have time to do any of this but a recent kernel must have fixed the issue.
I am running 4.18.9 now and the issue is gone.

Thank you for your help!