You are not logged in.
- Topics: Active | Unanswered
#1 2015-05-29 13:02:13
- ashleygc
- Member
- From: Europe
- Registered: 2014-06-23
- Posts: 12
- Website
Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
Hello all,
On 05 Apr 2015, I did a full upgrade (pacman -Syu), including a kernel update:
[2015-04-05 19:32] [ALPM] upgraded linux (3.18.6-1 -> 3.19.2-1)and since then I have been getting random crashes. I suspect an obscure kernel bug crept in somewhere after 3.18.6 (that version was working fine) and before (or in) 3.19.2. I have been doing full upgrades every week or so (and also upgrading Xen from AUR), in the hope that the bug would go away, but it hasn't.
I run a few Xen virtual machines, so there's obviously a lot of virtual memory coming and going, but I have run memtest86 successfully (one complete runthrough), so I do not suspect the memory hardware.
Looking in the journal for log messages, I see CPU register and stack trace dumps (see end note 1). I notice (end notes 2 and 3) that the function down_read_trylock (from linux/kernel/locking/rwsem.c) is in nearly all the crashes, and it's being called from the page_lock_anon_vma_read (from linux/mm/rmap.c) function (end note 4).
The file kernel/locking/rwsem.c has not changed between 3.18.6 and 3.19.2 (using the vanilla kernel sources from kernel.org), but mm/rmap.c has changed quite a bit. In particular, references to
mapping->i_mmap_mutexhave been replaced with
mapping->i_mmap_rwsemFor full diff, see end note 6.
In addition, taint info (see end note 5) shows the flags G=kernel, O=oot_module, and sometimes D=died (info from linux/kernel/panic.c and linux/include/linux/kernel.h). I expect to see G and D, but not O. I'm not sure if this indicates there's been a problem while loading/unloading a module. The current value of /proc/sys/kernel/tainted (on my currently running system) is 4096, which is 1<<12, which is the O=oot_module flag, so perhaps it can be ignored since it seems it's always present.
I'm a little out of my depth by now, having not done any kernel bug tracing or debuging before, so I guess my questions are:
Has anyone else been having the same or similar probems?
Might someone have some suggestions for how I should investigate further?
(Postscript: I've read How to Ask Questions the Smart Way, but if I've forgotten to do something obvious or post some obvious info, then apologies.)
End notes:
Here is a complete crash dump log. In this case, snmpd died, though the process that dies is not consistent (see end note 5)
May 29 11:23:46 hoom kernel: general protection fault: 0000 [#1] PREEMPT SMP May 29 11:23:46 hoom kernel: Modules linked in: uas usb_storage dm_crypt loop dm_mod tun xt_physdev br_netfilter fuse bridge stp llc xt_nat iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack it87 iptable_filter hwmon_vid ip_tables x_tables amdkfd amd_iommu_v2 crct10dif_pclmul crc32_pclmul crc32c_intel radeon ghash_clmulni_intel snd_hda_codec_via snd_hda_codec_hdmi snd_hda_codec_generic joydev aesni_intel ttm evdev snd_hda_intel aes_x86_64 mousedev lrw snd_hda_controller gf128mul glue_helper ablk_helper drm_kms_helper mac_hid snd_hda_codec psmouse cryptd r8168(O) snd_hwdep snd_pcm edac_core sp5100_tco edac_mce_amd drm snd_timer pcspkr serio_raw i2c_piix4 k10temp fam15h_power snd i2c_algo_bit tpm_infineon r8169 i2c_core mii soundcore May 29 11:23:46 hoom kernel: tpm_tis tpm shpchp button sch_fq_codel xen_acpi_processor processor xen_pciback xen_netback xen_blkback nfsd xen_gntalloc xen_gntdev xen_evtchn auth_rpcgss nfs oid_registry nfs_acl lockd fscache grace xenfs sunrpc xen_privcmd ext4 crc16 mbcache jbd2 sd_mod ata_generic pata_acpi hid_logitech_hidpp hid_logitech_dj usbhid hid atkbd libps2 pata_atiixp ahci libahci xhci_pci ohci_pci libata ehci_pci ohci_hcd ehci_hcd xhci_hcd usbcore scsi_mod usb_common i8042 serio May 29 11:23:46 hoom kernel: CPU: 6 PID: 578 Comm: snmpd Tainted: G O 4.0.4-2-ARCH #1 May 29 11:23:46 hoom kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-UD3P, BIOS F1 08/06/2013 May 29 11:23:46 hoom kernel: task: ffff880816a32880 ti: ffff88080cbfc000 task.ti: ffff88080cbfc000 May 29 11:23:46 hoom kernel: RIP: e030:[<ffffffff810bdc59>] [<ffffffff810bdc59>] down_read_trylock+0x9/0x30 May 29 11:23:46 hoom kernel: RSP: e02b:ffff88080cbff648 EFLAGS: 00010206 May 29 11:23:46 hoom kernel: RAX: 0000000000000000 RBX: ffff88010087a910 RCX: 0000000016191b1c May 29 11:23:46 hoom kernel: RDX: ffffffff8119ac30 RSI: ffff88080cbff6e8 RDI: 151515151515151d May 29 11:23:46 hoom kernel: RBP: ffff88080cbff648 R08: 0000000000017938 R09: 0000000000000002 May 29 11:23:46 hoom kernel: R10: ffff8807df6cc000 R11: 0000000000000003 R12: ffff88010087a911 May 29 11:23:46 hoom kernel: R13: ffffea0004322500 R14: 151515151515151d R15: ffffea0004322500 May 29 11:23:46 hoom kernel: FS: 00007f771616c700(0000) GS:ffff880820980000(0000) knlGS:0000000000000000 May 29 11:23:46 hoom kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b May 29 11:23:46 hoom kernel: CR2: 00007f77161a2000 CR3: 000000080c969000 CR4: 0000000000040660 May 29 11:23:46 hoom kernel: Stack: May 29 11:23:46 hoom kernel: ffff88080cbff678 ffffffff8119ac8e ffff88080cbff6e8 ffffea0004322500 May 29 11:23:46 hoom kernel: ffffea000b8e7480 ffff88010087a911 ffff88080cbff6d8 ffffffff8119aef3 May 29 11:23:46 hoom kernel: 0000000000000000 ffffea0004322580 ffffea000b8cb580 0000000000000000 May 29 11:23:46 hoom kernel: Call Trace: May 29 11:23:46 hoom kernel: [<ffffffff8119ac8e>] page_lock_anon_vma_read+0x5e/0x130 May 29 11:23:46 hoom kernel: [<ffffffff8119aef3>] rmap_walk+0x193/0x330 May 29 11:23:46 hoom kernel: [<ffffffff8119b2dc>] try_to_unmap+0x6c/0xb0 May 29 11:23:46 hoom kernel: [<ffffffff81199f10>] ? page_remove_rmap+0x120/0x120 May 29 11:23:46 hoom kernel: [<ffffffff81199360>] ? invalid_migration_vma+0x30/0x30 May 29 11:23:46 hoom kernel: [<ffffffff8119ac30>] ? page_get_anon_vma+0xc0/0xc0 May 29 11:23:46 hoom kernel: [<ffffffff81199330>] ? invalid_mkclean_vma+0x20/0x20 May 29 11:23:46 hoom kernel: [<ffffffff811c1237>] migrate_pages+0x4a7/0x910 May 29 11:23:46 hoom kernel: [<ffffffff811842d0>] ? pageblock_pfn_to_page+0xe0/0xe0 May 29 11:23:46 hoom kernel: [<ffffffff81185ca0>] ? isolate_freepages_block+0x3c0/0x3c0 May 29 11:23:46 hoom kernel: [<ffffffff8118706e>] compact_zone+0x4be/0x870 May 29 11:23:46 hoom kernel: [<ffffffff813859f2>] ? xen_send_IPI_one+0x32/0x70 May 29 11:23:46 hoom kernel: [<ffffffff8118748a>] compact_zone_order+0x6a/0x90 May 29 11:23:46 hoom kernel: [<ffffffff81187752>] try_to_compact_pages+0x102/0x280 May 29 11:23:46 hoom kernel: [<ffffffff81167793>] __alloc_pages_direct_compact+0x43/0x100 May 29 11:23:46 hoom kernel: [<ffffffff81167dc0>] __alloc_pages_nodemask+0x570/0x9f0 May 29 11:23:46 hoom kernel: [<ffffffff811682ba>] alloc_kmem_pages_node+0x7a/0x150 May 29 11:23:46 hoom kernel: [<ffffffff811b640c>] kmalloc_large_node+0x2c/0x60 May 29 11:23:46 hoom kernel: [<ffffffff811bd8c5>] __kmalloc_node_track_caller+0x275/0x320 May 29 11:23:46 hoom kernel: [<ffffffff814a72a6>] ? __netlink_dump_start+0x176/0x1b0 May 29 11:23:46 hoom kernel: [<ffffffff81464bd7>] ? __alloc_skb+0x87/0x210 May 29 11:23:46 hoom kernel: [<ffffffff81464af1>] __kmalloc_reserve.isra.5+0x31/0x90 May 29 11:23:46 hoom kernel: [<ffffffff81527fb0>] ? inet6_dump_ifmcaddr+0x20/0x20 May 29 11:23:46 hoom kernel: [<ffffffff81464bab>] ? __alloc_skb+0x5b/0x210 May 29 11:23:46 hoom kernel: [<ffffffff81464bd7>] __alloc_skb+0x87/0x210 May 29 11:23:46 hoom kernel: [<ffffffff814a4ed2>] netlink_alloc_skb+0x62/0x1e0 May 29 11:23:46 hoom kernel: [<ffffffff814a6a03>] netlink_dump+0xc3/0x2d0 May 29 11:23:46 hoom kernel: [<ffffffff814a6e7d>] netlink_recvmsg+0x26d/0x340 May 29 11:23:46 hoom kernel: [<ffffffff81459da2>] sock_recvmsg_nosec+0x52/0x80 May 29 11:23:46 hoom kernel: [<ffffffff8145c1dd>] SyS_recvfrom+0x11d/0x1a0 May 29 11:23:46 hoom kernel: [<ffffffff814587e6>] ? sock_alloc_file+0xa6/0x140 May 29 11:23:46 hoom kernel: [<ffffffff811f6bbd>] ? __fd_install+0x4d/0x70 May 29 11:23:46 hoom kernel: [<ffffffff811f6c05>] ? fd_install+0x25/0x30 May 29 11:23:46 hoom kernel: [<ffffffff8145b58f>] ? SyS_socket+0x9f/0xf0 May 29 11:23:46 hoom kernel: [<ffffffff8157a449>] system_call_fastpath+0x12/0x17 May 29 11:23:46 hoom kernel: Code: 6e bf 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 b7 a8 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 29 11:23:46 hoom kernel: RIP [<ffffffff810bdc59>] down_read_trylock+0x9/0x30 May 29 11:23:46 hoom kernel: RSP <ffff88080cbff648> May 29 11:23:46 hoom kernel: ---[ end trace 32e90ce275ac679e ]--- May 29 11:23:46 hoom systemd[1]: snmpd.service: main process exited, code=killed, status=11/SEGV May 29 11:23:46 hoom systemd[1]: Unit snmpd.service entered failed state. May 29 11:23:46 hoom systemd[1]: snmpd.service failed.Looking for IP in the journal:
journalctl >/tmp/journal ; grep ' kernel: IP: ' /tmp/journal | lessOutput:
Apr 06 11:00:02 hoom kernel: IP: [<ffffffff810bafc9>] down_read_trylock+0x9/0x30 Apr 06 11:00:03 hoom kernel: IP: [<ffffffff810bafc9>] down_read_trylock+0x9/0x30 Apr 06 11:00:04 hoom kernel: IP: [<ffffffff810bafc9>] down_read_trylock+0x9/0x30 Apr 16 08:32:26 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 15:44:20 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:52 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:18:18 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:18:18 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:18:19 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:20:00 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:20:00 hoom kernel: IP: [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 May 11 09:38:10 hoom kernel: IP: [<ffffffffa012805d>] gntdev_vma_find_special_page+0x1d/0x30 [xen_gntdev] May 13 17:46:39 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:45:00 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:45:27 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:49:39 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: IP: [<ffffffff81093840>] kthread_data+0x10/0x20 May 14 08:50:00 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: IP: [<ffffffff810bdb09>] down_read_trylock+0x9/0x30Looking for RIP in the journal:
journalctl >/tmp/journal ; grep ' kernel: RIP: ' /tmp/journal | lessOutput:
Apr 06 11:00:03 hoom kernel: RIP: e030:[<ffffffff810bafc9>] [<ffffffff810bafc9>] down_read_trylock+0x9/0x30 Apr 06 11:00:03 hoom kernel: RIP: e030:[<ffffffff810bafc9>] [<ffffffff810bafc9>] down_read_trylock+0x9/0x30 Apr 06 11:00:04 hoom kernel: RIP: e030:[<ffffffff810bafc9>] [<ffffffff810bafc9>] down_read_trylock+0x9/0x30 Apr 06 11:00:04 hoom kernel: RIP: e030:[<ffffffff810bafc9>] [<ffffffff810bafc9>] down_read_trylock+0x9/0x30 Apr 16 08:32:26 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 13:49:42 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 13:58:42 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:00:00 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:02:29 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:02:29 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:15:00 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:15:48 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:26:34 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:44:00 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 14:50:03 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 15:08:11 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 15:31:48 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 15:44:20 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 16:04:01 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 16:04:01 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 16 16:37:55 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:50 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:17:52 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:18:18 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:18:18 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:18:19 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:20:00 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 Apr 26 09:20:00 hoom kernel: RIP: e030:[<ffffffff810bb0a9>] [<ffffffff810bb0a9>] down_read_trylock+0x9/0x30 May 11 09:38:10 hoom kernel: RIP: e030:[<ffffffffa012805d>] [<ffffffffa012805d>] gntdev_vma_find_special_page+0x1d/0x30 [xen_gntdev] May 13 17:46:39 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:45:00 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:45:27 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:49:39 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: RIP: e030:[<ffffffff81093840>] [<ffffffff81093840>] kthread_data+0x10/0x20 May 14 08:50:00 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 14 08:50:00 hoom kernel: RIP: e030:[<ffffffff810bdb09>] [<ffffffff810bdb09>] down_read_trylock+0x9/0x30 May 21 21:05:28 hoom kernel: RIP: e030:[<ffffffff810bdc59>] [<ffffffff810bdc59>] down_read_trylock+0x9/0x30 May 22 09:45:38 hoom kernel: RIP: e030:[<ffffffff810bdc59>] [<ffffffff810bdc59>] down_read_trylock+0x9/0x30 May 22 09:57:35 hoom kernel: RIP: e030:[<ffffffff810bdc59>] [<ffffffff810bdc59>] down_read_trylock+0x9/0x30 May 22 09:57:55 hoom kernel: RIP: e030:[<ffffffff810bdc59>] [<ffffffff810bdc59>] down_read_trylock+0x9/0x30 May 29 11:23:46 hoom kernel: RIP: e030:[<ffffffff810bdc59>] [<ffffffff810bdc59>] down_read_trylock+0x9/0x30Looking for the function at the top of the call trace stack:
journalctl >/tmp/journal ; grep -A1 'Call Trace' /tmp/journal | grep 0x | lessOutput:
Apr 06 11:00:03 hoom kernel: [<ffffffff81196a3e>] page_lock_anon_vma_read+0x5e/0x130 Apr 06 11:00:03 hoom kernel: [<ffffffff81196a3e>] page_lock_anon_vma_read+0x5e/0x130 Apr 06 11:00:04 hoom kernel: [<ffffffff81196a3e>] page_lock_anon_vma_read+0x5e/0x130 Apr 06 11:00:04 hoom kernel: [<ffffffff81196a3e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 08:32:26 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 13:49:42 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 13:58:42 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 14:00:00 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 14:02:29 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 14:15:00 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 14:15:48 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 14:26:34 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 14:44:00 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 14:50:03 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 15:08:11 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 15:31:48 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 15:44:20 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 16:04:01 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 16:04:01 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 16 16:37:55 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:17:50 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:17:50 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:17:50 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:17:50 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:17:50 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:17:52 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:18:18 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:18:18 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:18:19 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:20:00 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 Apr 26 09:20:00 hoom kernel: [<ffffffff81196b9e>] page_lock_anon_vma_read+0x5e/0x130 May 11 09:38:10 hoom kernel: [<ffffffff8118ab4a>] ? vm_normal_page+0x7a/0xb0 May 11 09:38:10 hoom kernel: [<ffffffff8156de97>] dump_stack+0x4c/0x6e May 13 17:46:39 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 14 08:45:00 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 14 08:45:27 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 14 08:49:39 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 14 08:50:00 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 14 08:50:00 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 14 08:50:00 hoom kernel: [<ffffffff8108e595>] wq_worker_sleeping+0x15/0xa0 May 14 08:50:00 hoom kernel: [<ffffffff8156de97>] dump_stack+0x4c/0x6e May 14 08:50:00 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 14 08:50:00 hoom kernel: [<ffffffff811997ae>] page_lock_anon_vma_read+0x5e/0x130 May 21 21:05:28 hoom kernel: [<ffffffff8119ac5e>] page_lock_anon_vma_read+0x5e/0x130 May 22 09:45:38 hoom kernel: [<ffffffff8119ac5e>] page_lock_anon_vma_read+0x5e/0x130 May 22 09:57:35 hoom kernel: [<ffffffff8119ac5e>] page_lock_anon_vma_read+0x5e/0x130 May 22 09:57:55 hoom kernel: [<ffffffff8119ac5e>] page_lock_anon_vma_read+0x5e/0x130 May 22 10:00:39 hoom kernel: [<ffffffff81571be3>] dump_stack+0x4c/0x6e May 29 11:23:46 hoom kernel: [<ffffffff8119ac8e>] page_lock_anon_vma_read+0x5e/0x130The user-space process that dies is not consistent:
journalctl >/tmp/journal ; grep Tainted /tmp/journal | lessOutput:
Apr 06 11:00:02 hoom kernel: CPU: 2 PID: 2448 Comm: php Tainted: G O 3.19.2-1-ARCH #1 Apr 06 11:00:03 hoom kernel: CPU: 1 PID: 63 Comm: kswapd0 Tainted: G D O 3.19.2-1-ARCH #1 Apr 06 11:00:04 hoom kernel: CPU: 0 PID: 734 Comm: xfwm4 Tainted: G D O 3.19.2-1-ARCH #1 Apr 06 11:00:04 hoom kernel: CPU: 3 PID: 1135 Comm: thunderbird Tainted: G D O 3.19.2-1-ARCH #1 Apr 16 08:32:26 hoom kernel: CPU: 7 PID: 13402 Comm: Xvnc Tainted: G O 3.19.3-3-ARCH #1 Apr 16 13:49:42 hoom kernel: CPU: 4 PID: 26675 Comm: ssh Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 13:58:42 hoom kernel: CPU: 3 PID: 415 Comm: Xorg Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:00:00 hoom kernel: CPU: 1 PID: 24459 Comm: perl Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:02:29 hoom kernel: CPU: 5 PID: 551 Comm: snmpd Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:02:29 hoom kernel: CPU: 6 PID: 2 Comm: kthreadd Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:15:00 hoom kernel: CPU: 1 PID: 362 Comm: mysqld Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:15:48 hoom kernel: CPU: 3 PID: 732 Comm: gpg-agent Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:26:34 hoom kernel: CPU: 5 PID: 8598 Comm: panel-23-diskpe Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:44:00 hoom kernel: CPU: 2 PID: 25958 Comm: qemu-system-i38 Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 14:50:03 hoom kernel: CPU: 0 PID: 25403 Comm: rhythmbox Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 15:08:11 hoom kernel: CPU: 4 PID: 15301 Comm: qemu-dm Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 15:31:48 hoom kernel: CPU: 0 PID: 1533 Comm: qemu-dm Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 15:44:20 hoom kernel: CPU: 5 PID: 8595 Comm: panel-20-cpugra Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 16:04:01 hoom kernel: CPU: 2 PID: 1172 Comm: qemu-system-i38 Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 16:04:01 hoom kernel: CPU: 6 PID: 911 Comm: udisksd Tainted: G D O 3.19.3-3-ARCH #1 Apr 16 16:37:55 hoom kernel: CPU: 0 PID: 1685 Comm: qemu-dm Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:17:50 hoom kernel: CPU: 0 PID: 63 Comm: kswapd0 Tainted: G O 3.19.3-3-ARCH #1 Apr 26 09:17:50 hoom kernel: CPU: 7 PID: 11063 Comm: squid_redirect Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:17:50 hoom kernel: CPU: 4 PID: 2760 Comm: qemu-dm Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:17:50 hoom kernel: CPU: 5 PID: 11064 Comm: squid_redirect Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:17:50 hoom kernel: CPU: 7 PID: 435 Comm: Xorg Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:17:52 hoom kernel: CPU: 5 PID: 11056 Comm: DOM Worker Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:18:18 hoom kernel: CPU: 7 PID: 1515 Comm: rhythmbox Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:18:18 hoom kernel: CPU: 1 PID: 746 Comm: gpg-agent Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:18:19 hoom kernel: CPU: 4 PID: 579 Comm: snmpd Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:20:00 hoom kernel: CPU: 4 PID: 11076 Comm: php Tainted: G D O 3.19.3-3-ARCH #1 Apr 26 09:20:00 hoom kernel: CPU: 0 PID: 2591 Comm: qemu-dm Tainted: G D O 3.19.3-3-ARCH #1 May 11 09:38:10 hoom kernel: CPU: 0 PID: 17057 Comm: qemu-system-i38 Tainted: G O 4.0.1-1-ARCH #1 May 11 09:38:10 hoom kernel: CPU: 0 PID: 17057 Comm: qemu-system-i38 Tainted: G D O 4.0.1-1-ARCH #1 May 13 17:46:39 hoom kernel: CPU: 5 PID: 555 Comm: snmpd Tainted: G O 4.0.1-1-ARCH #1 May 14 08:45:00 hoom kernel: CPU: 7 PID: 63 Comm: kswapd0 Tainted: G O 4.0.1-1-ARCH #1 May 14 08:45:27 hoom kernel: CPU: 2 PID: 24007 Comm: qemu-dm Tainted: G D O 4.0.1-1-ARCH #1 May 14 08:49:39 hoom kernel: CPU: 1 PID: 567 Comm: snmpd Tainted: G D O 4.0.1-1-ARCH #1 May 14 08:50:00 hoom kernel: CPU: 3 PID: 117 Comm: kworker/3:1 Tainted: G D O 4.0.1-1-ARCH #1 May 14 08:50:00 hoom kernel: CPU: 1 PID: 22969 Comm: Xvnc Tainted: G D O 4.0.1-1-ARCH #1 May 14 08:50:00 hoom kernel: CPU: 3 PID: 117 Comm: kworker/3:1 Tainted: G D O 4.0.1-1-ARCH #1 May 14 08:50:00 hoom kernel: CPU: 3 PID: 117 Comm: kworker/3:1 Tainted: G D O 4.0.1-1-ARCH #1 May 14 08:50:00 hoom kernel: CPU: 7 PID: 24211 Comm: php Tainted: G D W O 4.0.1-1-ARCH #1 May 14 08:50:00 hoom kernel: CPU: 0 PID: 23626 Comm: panel-20-cpugra Tainted: G D W O 4.0.1-1-ARCH #1 May 21 21:05:28 hoom kernel: CPU: 1 PID: 28014 Comm: bacula-fd Tainted: G O 4.0.2-1-ARCH #1 May 22 09:45:38 hoom kernel: CPU: 0 PID: 589 Comm: snmpd Tainted: G O 4.0.2-1-ARCH #1 May 22 09:57:35 hoom kernel: CPU: 3 PID: 5667 Comm: qemu-dm Tainted: G D O 4.0.2-1-ARCH #1 May 22 09:57:55 hoom kernel: CPU: 1 PID: 4023 Comm: qemu-dm Tainted: G D O 4.0.2-1-ARCH #1 May 22 10:00:39 hoom kernel: CPU: 2 PID: 22103 Comm: java Tainted: G D O 4.0.2-1-ARCH #1 May 29 11:23:46 hoom kernel: CPU: 6 PID: 578 Comm: snmpd Tainted: G O 4.0.4-2-ARCH #1diff of rmap.c
diff linux-3.18.6/mm/rmap.c linux-3.19.1/mm/rmap.c |lessOutput:
26c26 < * mapping->i_mmap_mutex --- > * mapping->i_mmap_rwsem 74a75,76 > anon_vma->degree = 1; /* Reference for first vma */ > anon_vma->parent = anon_vma; 190a193,194 > /* vma reference or self-parent link for new root */ > anon_vma->degree++; 238a243,250 > * > * If dst->anon_vma is NULL this function tries to find and reuse existing > * anon_vma which has no vmas and only one child anon_vma. This prevents > * degradation of anon_vma hierarchy to endless linear chain in case of > * constantly forking task. On the other hand, an anon_vma with more than one > * child isn't reused even if there was no alive vma, thus rmap walker has a > * good chance of avoiding scanning the whole hierarchy when it searches where > * page is mapped. 258a271,282 > > /* > * Reuse existing anon_vma if its degree lower than two, > * that means it has no vma and only one anon_vma child. > * > * Do not chose parent anon_vma, otherwise first child > * will always reuse it. Root anon_vma is never reused: > * it has self-parent reference and at least one child. > */ > if (!dst->anon_vma && anon_vma != src->anon_vma && > anon_vma->degree < 2) > dst->anon_vma = anon_vma; 259a284,285 > if (dst->anon_vma) > dst->anon_vma->degree++; 282a309,311 > /* Drop inherited anon_vma, we'll reuse existing or allocate new. */ > vma->anon_vma = NULL; > 290a320,323 > /* An existing anon_vma has been reused, all done then. */ > if (vma->anon_vma) > return 0; > 303a337 > anon_vma->parent = pvma->anon_vma; 313a348 > anon_vma->parent->degree++; 344c379,380 < if (RB_EMPTY_ROOT(&anon_vma->rb_root)) --- > if (RB_EMPTY_ROOT(&anon_vma->rb_root)) { > anon_vma->parent->degree--; 345a382 > } 349a387,388 > if (vma->anon_vma) > vma->anon_vma->degree--; 359a399 > BUG_ON(anon_vma->degree); 586c626,627 < pmde = ACCESS_ONCE(*pmd); --- > pmde = *pmd; > barrier(); 1056c1097 < mem_cgroup_end_page_stat(memcg, locked, flags); --- > mem_cgroup_end_page_stat(memcg, &locked, &flags); 1086c1127 < mem_cgroup_end_page_stat(memcg, locked, flags); --- > mem_cgroup_end_page_stat(memcg, &locked, &flags); 1263c1304 < * we now hold anon_vma->rwsem or mapping->i_mmap_mutex. --- > * we now hold anon_vma->rwsem or mapping->i_mmap_rwsem. 1383c1424 < pteval = ptep_clear_flush(vma, address, pte); --- > pteval = ptep_clear_flush_notify(vma, address, pte); 1638c1679 < pgoff_t pgoff = page_to_pgoff(page); --- > pgoff_t pgoff; 1645a1687 > pgoff = page_to_pgoff(page); 1679c1721 < pgoff_t pgoff = page_to_pgoff(page); --- > pgoff_t pgoff; 1687c1729 < * so we can safely take mapping->i_mmap_mutex. --- > * so we can safely take mapping->i_mmap_rwsem. 1693c1735,1737 < mutex_lock(&mapping->i_mmap_mutex); --- > > pgoff = page_to_pgoff(page); > i_mmap_lock_read(mapping); 1714d1757 < 1716c1759 < mutex_unlock(&mapping->i_mmap_mutex); --- > i_mmap_unlock_read(mapping);In case there are some gurus reading this, here are all the Code lines from the crashes:
journalctl >/tmp/journal ; grep 'kernel: Code:' /tmp/journal | lessOutput:
Apr 06 11:00:03 hoom kernel: Code: de 6f 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 f7 58 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 06 11:00:03 hoom kernel: Code: de 6f 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 f7 58 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 06 11:00:04 hoom kernel: Code: de 6f 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 f7 58 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 06 11:00:04 hoom kernel: Code: de 6f 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 f7 58 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 08:32:26 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 13:49:42 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 13:58:42 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 14:00:00 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 14:02:29 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 14:15:00 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 14:15:48 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 14:26:34 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 14:44:00 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 14:50:03 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 15:08:11 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 15:31:48 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 15:44:20 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 16:04:01 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 16:04:02 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 16 16:37:55 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:17:50 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:17:50 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:17:50 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:17:50 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:17:50 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:17:52 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:18:18 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:18:18 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:18:19 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:20:00 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 Apr 26 09:20:00 hoom kernel: Code: be 70 4a 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 d7 59 4a 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 11 09:38:10 hoom kernel: Code: 5b 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 87 a8 00 00 00 55 48 89 e5 48 2b 70 68 48 8b 40 60 5d 48 c1 ee 0c <48> 8b 04 f0 c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 May 13 17:46:39 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 14 08:45:00 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 14 08:45:27 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 14 08:49:39 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 14 08:50:00 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 14 08:50:00 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 14 08:50:00 hoom kernel: Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 87 20 05 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 May 14 08:50:00 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 14 08:50:00 hoom kernel: Code: 3e 54 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 87 3d 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 21 21:05:28 hoom kernel: Code: 2e 90 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 77 79 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 22 09:45:38 hoom kernel: Code: 2e 90 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 77 79 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 22 09:57:35 hoom kernel: Code: 2e 90 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 77 79 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 22 09:57:55 hoom kernel: Code: 2e 90 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 77 79 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 May 29 11:23:46 hoom kernel: Code: 6e bf 4b 00 48 83 c4 18 5b 5d c3 0f 1f 80 00 00 00 00 48 89 45 e8 e8 b7 a8 4b 00 48 8b 45 e8 eb d8 90 66 66 66 66 90 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7
Offline
#2 2015-05-29 15:40:01
- ewaller
- Administrator
- From: Pasadena, CA
- Registered: 2009-07-13
- Posts: 20,612
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
A couple questions. The kernel was updated again on 4/8 and then on 4/13 it went to v4.0-1. We are now up to 4.04-1. Any reason you have not updated again?
Is it possible you are are being impacted by the changes in how processor microcode updates are applied?
Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
The shortest way to ruin a country is to give power to demagogues.— Dionysius of Halicarnassus
---
How to Ask Questions the Smart Way
Offline
#3 2015-05-29 16:27:37
- ashleygc
- Member
- From: Europe
- Registered: 2014-06-23
- Posts: 12
- Website
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
Thanks for the quick reply. To answer your questions:
Yes, I've been running pacman -Syu every now and again, including kernel updates. You can see the info hidden buried down in end note 5, but here's the journal log info more clearly:
Jan 24 20:52:32 hoom kernel: Linux version 3.18.2-2-ARCH (builduser@tobias) (gcc version 4.9.2 20141224 (prerelease) (GCC) ) #1 SMP PREEMPT Fri Jan 9 07:37:51 CET 2015 Mar 14 13:06:26 hoom kernel: Linux version 3.18.6-1-ARCH (builduser@tobias) (gcc version 4.9.2 20141224 (prerelease) (GCC) ) #1 SMP PREEMPT Sat Feb 7 08:44:05 CET 2015 Apr 06 10:14:39 hoom kernel: Linux version 3.19.2-1-ARCH (builduser@tobias) (gcc version 4.9.2 20150304 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Mar 18 16:21:02 CET 2015 Apr 08 22:52:57 hoom kernel: Linux version 3.19.3-3-ARCH (builduser@tobias) (gcc version 4.9.2 20150304 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 8 14:10:00 CEST 2015 May 09 21:35:07 hoom kernel: Linux version 4.0.1-1-ARCH (builduser@tobias) (gcc version 5.1.0 (GCC) ) #1 SMP PREEMPT Wed Apr 29 12:00:26 CEST 2015 May 14 20:52:23 hoom kernel: Linux version 4.0.2-1-ARCH (builduser@tobias) (gcc version 5.1.0 (GCC) ) #1 SMP PREEMPT Thu May 7 06:47:54 CEST 2015 May 23 22:42:18 hoom kernel: Linux version 4.0.4-2-ARCH (builduser@foutrelis) (gcc version 5.1.0 (GCC) ) #1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015I have an AMD processor (vendor_id: AuthenticAMD, cpu family: 21 (0x15), model: 2, model name: AMD FX(tm)-8350 Eight-Core Processor, stepping: 0, microcode: 0x6000822). It looks like microcode updates from 0x06000822 to 0x06000832 have been failing:
Jan 03 21:41:05 hoom kernel: microcode: CPU0: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: CPU1: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: CPU2: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: CPU3: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: CPU4: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: CPU5: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: CPU6: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: CPU7: patch_level=0x06000822 Jan 03 21:41:05 hoom kernel: microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba Jan 03 21:41:05 hoom kernel: microcode: CPU0: update failed for patch_level=0x06000832 Jan 03 21:41:05 hoom kernel: microcode: CPU1: update failed for patch_level=0x06000832 Jan 03 21:41:05 hoom kernel: microcode: CPU2: update failed for patch_level=0x06000832 Jan 03 21:41:05 hoom kernel: microcode: CPU3: update failed for patch_level=0x06000832 Jan 03 21:41:05 hoom kernel: microcode: CPU4: update failed for patch_level=0x06000832 Jan 03 21:41:05 hoom kernel: microcode: CPU5: update failed for patch_level=0x06000832 Jan 03 21:41:05 hoom kernel: microcode: CPU6: update failed for patch_level=0x06000832 Jan 03 21:41:05 hoom kernel: microcode: CPU7: update failed for patch_level=0x06000832Since Jan 18th, I have seen no more messages like above. The only microcode lines I see in the journal are:
Jan 18 16:21:15 hoom kernel: [drm] Loading CAICOS Microcodebut I assume they're not relevant and are for the graphics card:
[ashley@hoom ~]$ lspci | grep -i caicos 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
I have the package linux-firmware installed (version 20150511.8e18132-1) and the MD5 checksums for all the files in /usr/lib/firmware/amd-ucode/ match the checksums from the files I downloaded from http://www.amd64.org/microcode.html.
I'll look forther into getting the microcode updated.
Offline
#4 2015-05-29 17:09:13
- ashleygc
- Member
- From: Europe
- Registered: 2014-06-23
- Posts: 12
- Website
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
I have just found this Arch forum post: [SOLVED] AMD Microcode update failed (on Xen host). I'll follow its advice and report back.
Offline
#5 2015-05-29 18:57:34
- ashleygc
- Member
- From: Europe
- Registered: 2014-06-23
- Posts: 12
- Website
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
Ok, I've followed the advice from the Arch forum post, copying the microcode file (microcode_amd_fam15h.bin for me) into /boot, updating /etc/xen/grub.conf and /etc/grub.d/09_xen, running grub-mkconfig, rebooting. Xen has updated the microcode from 0x6000822 to 0x6000832:
[ashley@hoom ~]$ cat /proc/cpuinfo
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0
microcode : 0x6000832
...I'll wait and see if I continue getting random crashes. If I get none after 2 weeks or so, I'll mark this one as solved.
Offline
#6 2015-05-30 06:29:19
- ashleygc
- Member
- From: Europe
- Registered: 2014-06-23
- Posts: 12
- Website
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
Unfortunately, I had another crash, same as before (IP:down_read_trylock+0x9/0x30 and Call Trace: page_lock_anon_vma_read+0x5e/0x130).
Time to compile a custom kernel with all the debugging options turned on and see what I can see...
Offline
#7 2015-07-17 09:30:49
- vollekannehoschi
- Member
- Registered: 2015-03-08
- Posts: 29
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
did you solved your problem since your last post?
Offline
#8 2015-07-17 16:14:11
- ashleygc
- Member
- From: Europe
- Registered: 2014-06-23
- Posts: 12
- Website
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
I did not solve the problem. Instead, I switched to the linux-lts (long-term support) kernel, and the problem has gone away (current time without crashes - 18 days), thus demonstrating that it's not dodgy hardware.
Offline
#9 2015-12-13 20:37:57
- ashleygc
- Member
- From: Europe
- Registered: 2014-06-23
- Posts: 12
- Website
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
Well, it's 5 months later, and I recently updated the linux-lts package:
[2015-12-09 09:56] [ALPM] upgraded linux-lts (3.14.52-1 -> 4.1.13-1)and now I'm getting the same crashes as before (down_read_trylock+0x9/0x30, and page_lock_anon_vma_read+0x50/0x100 at the top of the call trace).
Has anyone else seen anything similar?
Last edited by ashleygc (2015-12-13 20:38:21)
Offline
#10 2015-12-14 10:41:03
- vollekannehoschi
- Member
- Registered: 2015-03-08
- Posts: 29
Re: Random crashes in down_read_trylock, since 3.18.6<kernelver<=3.19.2
I send my PC back to the vendor. They checked it and made a performace/over night test with Windows (unfortunately) and got no problems. I had not the time to plug it in again 
I check back when I do so 
Offline