You are not logged in.
Pages: 1
Hi,
I am running a home server (Owncloud on LAMP, Samba, Logitech Media Server (AUR)), which has been running without issues for months. A week ago, the server started having out of memory issues. The OOM Killer keeps killing slimserver.pl (Logitech Media Server), mariadb, and httpd. I first noticed this because the media server was down; outwardly, Owncloud seems to run normally, as mariadb and httpd get restarted after they are killed.
I have no idea what is causing this issue. slimserver.pl usually seems to get killed first, but the issues do not stop afterwards, so I think it is a victim rather than the culprit.
The server has 8GB physical memory and a swap partition that hardly seems to get used. This is what free -M gives me (OOM has killed a process a few minutes ago):
total used free shared buff/cache available
Mem: 7986 425 230 0 7331 7262
Swap: 6143 56 6087
Very little memory is actually listed as 'used', while plenty is available. As I understand, the low 'free' value is due to the fact that the OS uses most of the free space for caching, and thus not relevant here.
At the same time, this is the output of top. Here, a lot of virtual memory seems to be committed (mostly to httpd), but it does not actually seem to get 'used' yet:
top - 10:22:22 up 13:05, 1 user, load average: 0,07, 0,15, 0,29
Tasks: 233 total, 1 running, 232 sleeping, 0 stopped, 0 zombie
%Cpu0 : 4,0/0,7 5
%Cpu1 : 6,6/0,7 7
GiB Mem : 9,1/7,800
GiB Swap: 0,9/6,000
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
1 root 20 0 195,9m 4,5m 0,0 0,1 0:00.63 S systemd
204 root 20 0 88,2m 10,0m 0,0 0,1 0:00.99 S `- systemd-journal
238 root 20 0 35,8m 1,9m 0,0 0,0 0:00.10 S `- systemd-udevd
428 systemd+ 20 0 111,5m 2,0m 0,0 0,0 0:00.02 S `- systemd-timesyn
433 root 20 0 14,2m 2,4m 0,0 0,0 0:00.05 S `- crond
435 dbus 20 0 32,2m 2,5m 0,0 0,0 0:00.06 S `- dbus-daemon
447 root 20 0 37,6m 2,7m 0,0 0,0 0:00.04 S `- systemd-logind
460 root 20 0 36,6m 2,7m 0,0 0,0 0:11.13 S `- x2gocleansessio
579 root 20 0 6,6m 1,6m 0,0 0,0 0:01.27 S `- dhcpcd
591 root 20 0 185,5m 3,6m 0,0 0,0 0:00.00 S `- sddm
623 root 20 0 212,7m 5,5m 0,0 0,1 0:11.83 S `- Xorg
740 root 20 0 143,3m 4,0m 0,0 0,1 0:00.00 S `- sddm-helper
748 sddm 20 0 580,2m 13,4m 0,0 0,2 2:40.12 S `- sddm-greeter
602 root 20 0 32,8m 6,0m 0,0 0,1 0:04.53 S `- ddclient - slee
603 root 20 0 26,6m 0,0m 0,0 0,0 0:00.00 S `- xrdp-sesman
701 root 20 0 229,5m 3,2m 0,0 0,0 0:02.67 S `- nmbd
725 root 20 0 311,7m 4,3m 0,0 0,1 0:00.07 S `- smbd
726 root 20 0 308,0m 1,8m 0,0 0,0 0:00.00 S `- smbd-notifyd
727 root 20 0 308,0m 2,5m 0,0 0,0 0:00.00 S `- cleanupd
742 root 20 0 311,7m 2,7m 0,0 0,0 0:00.01 S `- lpqd
743 sddm 20 0 53,4m 3,6m 0,0 0,0 0:00.03 S `- systemd
744 sddm 20 0 96,4m 0,0m 0,0 0,0 0:00.00 S `- (sd-pam)
768 sddm 20 0 32,0m 1,6m 0,0 0,0 0:00.00 S `- dbus-daemon
4373 root 20 0 327,1m 9,6m 0,0 0,1 0:00.67 S `- httpd
4374 http 20 0 338,7m 23,8m 0,0 0,3 0:35.46 S `- httpd
4376 http 20 0 338,7m 24,1m 0,0 0,3 0:39.48 S `- httpd
7192 http 20 0 338,7m 23,4m 0,0 0,3 0:29.88 S `- httpd
7213 http 20 0 338,7m 23,8m 0,0 0,3 0:37.98 S `- httpd
15096 http 20 0 338,7m 24,7m 0,0 0,3 0:25.32 S `- httpd
15097 http 20 0 338,7m 25,5m 0,0 0,3 0:24.02 S `- httpd
24381 http 20 0 338,7m 27,4m 0,0 0,3 0:08.96 S `- httpd
25150 http 20 0 338,7m 28,8m 0,0 0,4 0:05.15 S `- httpd
658 http 20 0 338,6m 27,4m 0,0 0,3 0:02.33 S `- httpd
8623 http 20 0 334,6m 23,0m 0,0 0,3 0:00.51 S `- httpd
24361 root 20 0 97,8m 6,2m 0,0 0,1 0:00.00 S `- sshd
24379 cube 20 0 97,8m 3,8m 0,0 0,0 0:00.66 S `- sshd
24380 cube 20 0 6,2m 1,4m 0,0 0,0 0:00.00 S `- sftp-server
1362 cube 20 0 17,1m 4,0m 0,0 0,1 0:00.01 S `- bash
10792 cube 20 0 37,9m 4,2m 0,7 0,1 0:00.07 R `- top
24372 cube 20 0 53,4m 4,9m 0,0 0,1 0:00.01 S `- systemd
24374 cube 20 0 176,4m 1,3m 0,0 0,0 0:00.00 S `- (sd-pam)
6295 mysql 20 0 760,3m 123,5m 0,0 1,5 0:00.73 S `- mysqld
And this is the section from the log when the OOM killer first becomes active. The server was rebooted 13 hours ago, ran normally, then 1 hour ago the first process was killed:
Sep 27 09:19:00 agora kernel: x2golistsession invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), order=2, oom_score_adj=0
Sep 27 09:19:00 agora kernel: x2golistsession cpuset=/ mems_allowed=0
Sep 27 09:19:00 agora kernel: CPU: 1 PID: 26540 Comm: x2golistsession Not tainted 4.7.4-1-ARCH #1
Sep 27 09:19:00 agora kernel: Hardware name: System manufacturer System Product Name/P5Q3, BIOS 1001 03/10/2010
Sep 27 09:19:00 agora kernel: 0000000000000286 000000008de34b4a ffff8801bc387b48 ffffffff812ecb02
Sep 27 09:19:00 agora kernel: ffff8801bc387d28 ffff8800c668dcc0 ffff8801bc387bb8 ffffffff811f6f5c
Sep 27 09:19:00 agora kernel: ffffffff81178a20 ffff880100000000 ffffffff810fdbe9 ffffffff00000000
Sep 27 09:19:00 agora kernel: Call Trace:
Sep 27 09:19:00 agora kernel: [<ffffffff812ecb02>] dump_stack+0x63/0x81
Sep 27 09:19:00 agora kernel: [<ffffffff811f6f5c>] dump_header+0x60/0x1e8
Sep 27 09:19:00 agora kernel: [<ffffffff81178a20>] ? page_alloc_cpu_notify+0x50/0x50
Sep 27 09:19:00 agora kernel: [<ffffffff810fdbe9>] ? smp_call_function_single+0xd9/0x140
Sep 27 09:19:00 agora kernel: [<ffffffff8117635a>] oom_kill_process+0x22a/0x440
Sep 27 09:19:00 agora kernel: [<ffffffff811769ca>] out_of_memory+0x40a/0x4b0
Sep 27 09:19:00 agora kernel: [<ffffffff813017e8>] ? find_next_bit+0x18/0x20
Sep 27 09:19:00 agora kernel: [<ffffffff8117c0bb>] __alloc_pages_nodemask+0xf0b/0xf30
Sep 27 09:19:00 agora kernel: [<ffffffff8117c434>] alloc_kmem_pages_node+0x54/0xd0
Sep 27 09:19:00 agora kernel: [<ffffffff81077be6>] copy_process.part.8+0x136/0x19b0
Sep 27 09:19:00 agora kernel: [<ffffffff811d745c>] ? kmem_cache_alloc+0x1bc/0x1e0
Sep 27 09:19:00 agora kernel: [<ffffffff811fc69c>] ? get_empty_filp+0x5c/0x1c0
Sep 27 09:19:00 agora kernel: [<ffffffff811fc716>] ? get_empty_filp+0xd6/0x1c0
Sep 27 09:19:00 agora kernel: [<ffffffff81079637>] _do_fork+0xd7/0x3d0
Sep 27 09:19:00 agora kernel: [<ffffffff810799d9>] SyS_clone+0x19/0x20
Sep 27 09:19:00 agora kernel: [<ffffffff81003c07>] do_syscall_64+0x57/0xb0
Sep 27 09:19:00 agora kernel: [<ffffffff815d7821>] entry_SYSCALL64_slow_path+0x25/0x25
Sep 27 09:19:00 agora kernel: Mem-Info:
Sep 27 09:19:00 agora kernel: active_anon:53730 inactive_anon:59987 isolated_anon:0
active_file:850438 inactive_file:253521 isolated_file:0
unevictable:0 dirty:45201 writeback:0 unstable:0
slab_reclaimable:730994 slab_unreclaimable:28648
mapped:14632 shmem:109 pagetables:3665 bounce:0
free:45387 free_pcp:29 free_cma:0
Sep 27 09:19:00 agora kernel: Node 0 DMA free:15896kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:
Sep 27 09:19:00 agora kernel: lowmem_reserve[]: 0 3229 7964 7964
Sep 27 09:19:00 agora kernel: Node 0 DMA32 free:103732kB min:27352kB low:34188kB high:41024kB active_anon:72976kB inactive_anon:93756kB active_file:1340868kB inactive_file:448348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3389
Sep 27 09:19:00 agora kernel: lowmem_reserve[]: 0 0 4734 4734
Sep 27 09:19:00 agora kernel: Node 0 Normal free:61920kB min:40096kB low:50120kB high:60144kB active_anon:141944kB inactive_anon:146192kB active_file:2060884kB inactive_file:565736kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:49
Sep 27 09:19:00 agora kernel: lowmem_reserve[]: 0 0 0 0
Sep 27 09:19:00 agora kernel: Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Sep 27 09:19:00 agora kernel: Node 0 DMA32: 10082*4kB (UME) 7940*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 103848kB
Sep 27 09:19:00 agora kernel: Node 0 Normal: 15243*4kB (UM) 116*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61900kB
Sep 27 09:19:00 agora kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 27 09:19:00 agora kernel: 1106473 total pagecache pages
Sep 27 09:19:00 agora kernel: 2374 pages in swap cache
Sep 27 09:19:00 agora kernel: Swap cache stats: add 49537, delete 47163, find 183880/186643
Sep 27 09:19:00 agora kernel: Free swap = 6144904kB
Sep 27 09:19:00 agora kernel: Total swap = 6291452kB
Sep 27 09:19:00 agora kernel: 2096653 pages RAM
Sep 27 09:19:00 agora kernel: 0 pages HighMem/MovableOnly
Sep 27 09:19:00 agora kernel: 52027 pages reserved
Sep 27 09:19:00 agora kernel: 0 pages hwpoisoned
Sep 27 09:19:00 agora kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Sep 27 09:19:00 agora kernel: [ 204] 0 204 22582 3535 40 3 78 0 systemd-journal
Sep 27 09:19:00 agora kernel: [ 238] 0 238 9169 481 21 3 373 -1000 systemd-udevd
Sep 27 09:19:00 agora kernel: [ 428] 192 428 28539 510 26 4 99 0 systemd-timesyn
Sep 27 09:19:00 agora kernel: [ 433] 0 433 3643 626 13 3 114 0 crond
Sep 27 09:19:00 agora kernel: [ 435] 81 435 8251 698 20 3 63 -900 dbus-daemon
Sep 27 09:19:00 agora kernel: [ 447] 0 447 9631 814 22 3 111 0 systemd-logind
Sep 27 09:19:00 agora kernel: [ 460] 0 460 9365 701 23 3 1008 0 x2gocleansessio
Sep 27 09:19:00 agora kernel: [ 579] 0 579 1685 397 9 3 52 0 dhcpcd
Sep 27 09:19:00 agora kernel: [ 586] 996 586 87469 57974 176 3 986 0 slimserver.pl
Sep 27 09:19:00 agora kernel: [ 591] 0 591 47499 917 47 3 328 0 sddm
Sep 27 09:19:00 agora kernel: [ 602] 0 602 8387 1773 19 3 457 0 ddclient - slee
Sep 27 09:19:00 agora kernel: [ 603] 0 603 6812 0 16 3 149 0 xrdp-sesman
Sep 27 09:19:00 agora kernel: [ 623] 0 623 54440 1416 101 3 1792 0 Xorg
Sep 27 09:19:00 agora kernel: [ 688] 89 688 196239 16152 131 4 21369 0 mysqld
Sep 27 09:19:00 agora kernel: [ 701] 0 701 58759 838 109 4 422 0 nmbd
Sep 27 09:19:00 agora kernel: [ 725] 0 725 79798 1136 151 4 599 0 smbd
Sep 27 09:19:00 agora kernel: [ 726] 0 726 78841 458 144 4 613 0 smbd-notifyd
Sep 27 09:19:00 agora kernel: [ 727] 0 727 78845 607 143 4 613 0 cleanupd
Sep 27 09:19:00 agora kernel: [ 740] 0 740 36681 1040 64 4 328 0 sddm-helper
Sep 27 09:19:00 agora kernel: [ 742] 0 742 79794 705 146 4 593 0 lpqd
Sep 27 09:19:00 agora kernel: [ 743] 997 743 13670 1095 30 3 150 0 systemd
Sep 27 09:19:00 agora kernel: [ 744] 997 744 24688 9 49 3 419 0 (sd-pam)
Sep 27 09:19:00 agora kernel: [ 748] 997 748 148414 3297 169 3 4943 0 sddm-greeter
Sep 27 09:19:00 agora kernel: [ 768] 997 768 8186 417 19 3 80 0 dbus-daemon
Sep 27 09:19:00 agora kernel: [ 4373] 0 4373 83725 2486 138 3 1513 0 httpd
Sep 27 09:19:00 agora kernel: [ 4374] 33 4374 86718 6065 140 3 818 0 httpd
Sep 27 09:19:00 agora kernel: [ 4375] 33 4375 86716 5939 140 3 812 0 httpd
Sep 27 09:19:00 agora kernel: [ 4376] 33 4376 86716 6155 140 3 807 0 httpd
Sep 27 09:19:00 agora kernel: [ 7192] 33 7192 86718 5986 140 3 819 0 httpd
Sep 27 09:19:00 agora kernel: [ 7213] 33 7213 86717 5939 140 3 812 0 httpd
Sep 27 09:19:00 agora kernel: [15096] 33 15096 86717 6409 140 3 774 0 httpd
Sep 27 09:19:00 agora kernel: [15097] 33 15097 86670 5984 139 3 779 0 httpd
Sep 27 09:19:00 agora kernel: [24361] 0 24361 25045 1703 52 3 0 0 sshd
Sep 27 09:19:00 agora kernel: [24372] 1000 24372 13670 1504 30 3 0 0 systemd
Sep 27 09:19:00 agora kernel: [24374] 1000 24374 45170 333 52 3 111 0 (sd-pam)
Sep 27 09:19:00 agora kernel: [24379] 1000 24379 25045 851 49 3 0 0 sshd
Sep 27 09:19:00 agora kernel: [24380] 1000 24380 1590 382 9 3 0 0 sftp-server
Sep 27 09:19:00 agora kernel: [24381] 33 24381 86670 6704 139 3 720 0 httpd
Sep 27 09:19:00 agora kernel: [25150] 33 25150 85134 5358 136 3 737 0 httpd
Sep 27 09:19:00 agora kernel: [25210] 33 25210 85134 5353 136 3 737 0 httpd
Sep 27 09:19:00 agora kernel: [25560] 1001 25560 82889 5024 157 4 329 0 smbd
Sep 27 09:19:00 agora kernel: [26539] 0 26539 4319 860 13 3 0 0 sh
Sep 27 09:19:00 agora kernel: [26540] 0 26540 13129 3447 31 3 0 0 x2golistsession
Sep 27 09:19:00 agora kernel: Out of memory: Kill process 586 (slimserver.pl) score 16 or sacrifice child
Sep 27 09:19:00 agora kernel: Killed process 586 (slimserver.pl) total-vm:349876kB, anon-rss:224640kB, file-rss:7256kB, shmem-rss:0kB
Sep 27 09:19:00 agora systemd[1]: logitechmediaserver.service: Main process exited, code=killed, status=9/KILL
Sep 27 09:19:00 agora systemd[1]: logitechmediaserver.service: Unit entered failed state.
Sep 27 09:19:00 agora systemd[1]: logitechmediaserver.service: Failed with result 'signal'.
Can anybody give me a pointer on where to start looking based on this? I do not even understand which type of memory is depleted - the process list does not look fundamentally different from the output of top, i.e. I do not see a process doing anything unusual.
Thank you!
Offline
There is a bug with kernel 4.7.x and OOM killer. As far as I know it has been fixed with version 4.7.5. Which kernel version do you have?
Offline
Thanks fturco, that is probably it. I'm on the core repository, so currently kernel 4.7.4-1.
This post then seems to discuss the same issue: https://bbs.archlinux.org/viewtopic.php?id=216716
A patch concerning this was discussed on LKML and seems to have been included in version 4.7.5-1, so I suppose I will just wait for this to move into core.
Offline
You can use this script to see who is using all the RAM (it lists nicely beginning with the less RAM eater to the most calculating also private + shared :
#!/usr/bin/env python
import getopt
import time
import errno
import os
import sys
try:
# md5 module is deprecated on python 2.6
# so try the newer hashlib first
import hashlib
md5_new = hashlib.md5
except ImportError:
import md5
md5_new = md5.new
# The following exits cleanly on Ctrl-C or EPIPE
# while treating other exceptions as before.
def std_exceptions(etype, value, tb):
sys.excepthook = sys.__excepthook__
if issubclass(etype, KeyboardInterrupt):
pass
elif issubclass(etype, IOError) and value.errno == errno.EPIPE:
pass
else:
sys.__excepthook__(etype, value, tb)
sys.excepthook = std_exceptions
#
# Define some global variables
#
uname = os.uname()
if uname[0] == "FreeBSD":
proc = "/compat/linux/proc/"
else:
proc = "/proc/"
PAGESIZE = os.sysconf("SC_PAGE_SIZE") / 1024 #KiB
our_pid = os.getpid()
have_pss = 0
#
# Functions
#
def parse_options():
try:
long_options = ['split-args', 'help']
opts, args = getopt.getopt(sys.argv[1:], "shp:w:", long_options)
except getopt.GetoptError, e:
print help()
sys.exit(3)
# ps_mem.py options
split_args = False
pids_to_show = None
watch = None
for o, a in opts:
if o in ('-s', '--split-args'):
split_args = True
if o in ('-h', '--help'):
print help()
sys.exit(0)
if o in ('-p',):
try:
pids_to_show = [int(x) for x in a.split(',')]
except:
print help()
sys.exit(3)
if o in ('-w',):
try:
watch = int(a)
except:
print help()
sys.exit(3)
return (split_args, pids_to_show, watch)
def help():
help_msg = 'ps_mem.py - Show process memory usage\n'\
'\n'\
'-h Show this help\n'\
'-w <N> Measure and show process memory every N seconds\n'\
'-p <pid>[,pid2,...pidN] Only show memory usage PIDs in the specified list\n'
return help_msg
#(major,minor,release)
def kernel_ver():
kv = open(proc + "sys/kernel/osrelease", "rt").readline().split(".")[:3]
last = len(kv)
if last == 2:
kv.append('0')
last -= 1
for char in "-_":
kv[last] = kv[last].split(char)[0]
try:
int(kv[last])
except:
kv[last] = 0
return (int(kv[0]), int(kv[1]), int(kv[2]))
#return Private,Shared
#Note shared is always a subset of rss (trs is not always)
def getMemStats(pid):
global have_pss
mem_id = pid #unique
Private_lines = []
Shared_lines = []
Pss_lines = []
Rss = (int(open(proc + str(pid) + "/statm", "rt").readline().split()[1])
* PAGESIZE)
if os.path.exists(proc + str(pid) + "/smaps"): #stat
digester = md5_new()
for line in open(proc + str(pid) + "/smaps", "rb").readlines(): #open
# Note we checksum smaps as maps is usually but
# not always different for separate processes.
digester.update(line)
line = line.decode("ascii")
if line.startswith("Shared"):
Shared_lines.append(line)
elif line.startswith("Private"):
Private_lines.append(line)
elif line.startswith("Pss"):
have_pss = 1
Pss_lines.append(line)
mem_id = digester.hexdigest()
Shared = sum([int(line.split()[1]) for line in Shared_lines])
Private = sum([int(line.split()[1]) for line in Private_lines])
#Note Shared + Private = Rss above
#The Rss in smaps includes video card mem etc.
if have_pss:
pss_adjust = 0.5 # add 0.5KiB as this avg error due to trunctation
Pss = sum([float(line.split()[1])+pss_adjust for line in Pss_lines])
Shared = Pss - Private
elif (2,6,1) <= kernel_ver() <= (2,6,9):
Shared = 0 #lots of overestimation, but what can we do?
Private = Rss
else:
Shared = int(open(proc+str(pid)+"/statm", "rt").readline().split()[2])
Shared *= PAGESIZE
Private = Rss - Shared
return (Private, Shared, mem_id)
def getCmdName(pid, split_args):
cmdline = open(proc+"%d/cmdline" % pid, "rt").read().split("\0")
if cmdline[-1] == '' and len(cmdline) > 1:
cmdline = cmdline[:-1]
path = os.path.realpath(proc+"%d/exe" % pid) #exception for kernel threads
if split_args:
return " ".join(cmdline)
if path.endswith(" (deleted)"):
path = path[:-10]
if os.path.exists(path):
path += " [updated]"
else:
#The path could be have prelink stuff so try cmdline
#which might have the full path present. This helped for:
#/usr/libexec/notification-area-applet.#prelink#.fX7LCT (deleted)
if os.path.exists(cmdline[0]):
path = cmdline[0] + " [updated]"
else:
path += " [deleted]"
exe = os.path.basename(path)
cmd = open(proc+"%d/status" % pid, "rt").readline()[6:-1]
if exe.startswith(cmd):
cmd = exe #show non truncated version
#Note because we show the non truncated name
#one can have separated programs as follows:
#584.0 KiB + 1.0 MiB = 1.6 MiB mozilla-thunder (exe -> bash)
# 56.0 MiB + 22.2 MiB = 78.2 MiB mozilla-thunderbird-bin
return cmd
#The following matches "du -h" output
#see also human.py
def human(num, power="Ki"):
powers = ["Ki", "Mi", "Gi", "Ti"]
while num >= 1000: #4 digits
num /= 1024.0
power = powers[powers.index(power)+1]
return "%.1f %s" % (num, power)
def cmd_with_count(cmd, count):
if count > 1:
return "%s (%u)" % (cmd, count)
else:
return cmd
#Warn of possible inaccuracies
#2 = accurate & can total
#1 = accurate only considering each process in isolation
#0 = some shared mem not reported
#-1= all shared mem not reported
def shared_val_accuracy():
"""http://wiki.apache.org/spamassassin/TopSharedMemoryBug"""
kv = kernel_ver()
if kv[:2] == (2,4):
if open(proc+"meminfo", "rt").read().find("Inact_") == -1:
return 1
return 0
elif kv[:2] == (2,6):
pid = str(os.getpid())
if os.path.exists(proc+pid+"/smaps"):
if open(proc+pid+"/smaps", "rt").read().find("Pss:")!=-1:
return 2
else:
return 1
if (2,6,1) <= kv <= (2,6,9):
return -1
return 0
elif kv[0] > 2:
return 2
else:
return 1
def show_shared_val_accuracy( possible_inacc ):
if possible_inacc == -1:
sys.stderr.write(
"Warning: Shared memory is not reported by this system.\n"
)
sys.stderr.write(
"Values reported will be too large, and totals are not reported\n"
)
elif possible_inacc == 0:
sys.stderr.write(
"Warning: Shared memory is not reported accurately by this system.\n"
)
sys.stderr.write(
"Values reported could be too large, and totals are not reported\n"
)
elif possible_inacc == 1:
sys.stderr.write(
"Warning: Shared memory is slightly over-estimated by this system\n"
"for each program, so totals are not reported.\n"
)
sys.stderr.close()
def get_memory_usage( pids_to_show, split_args, include_self=False, only_self=False ):
cmds = {}
shareds = {}
mem_ids = {}
count = {}
for pid in os.listdir(proc):
if not pid.isdigit():
continue
pid = int(pid)
# Some filters
if only_self and pid != our_pid:
continue
if pid == our_pid and not include_self:
continue
if pids_to_show is not None and pid not in pids_to_show:
continue
try:
cmd = getCmdName(pid, split_args)
except:
#permission denied or
#kernel threads don't have exe links or
#process gone
continue
try:
private, shared, mem_id = getMemStats(pid)
except:
continue #process gone
if shareds.get(cmd):
if have_pss: #add shared portion of PSS together
shareds[cmd] += shared
elif shareds[cmd] < shared: #just take largest shared val
shareds[cmd] = shared
else:
shareds[cmd] = shared
cmds[cmd] = cmds.setdefault(cmd, 0) + private
if cmd in count:
count[cmd] += 1
else:
count[cmd] = 1
mem_ids.setdefault(cmd, {}).update({mem_id:None})
#Add shared mem for each program
total = 0
for cmd in cmds:
cmd_count = count[cmd]
if len(mem_ids[cmd]) == 1 and cmd_count > 1:
# Assume this program is using CLONE_VM without CLONE_THREAD
# so only account for one of the processes
cmds[cmd] /= cmd_count
if have_pss:
shareds[cmd] /= cmd_count
cmds[cmd] = cmds[cmd] + shareds[cmd]
total += cmds[cmd] #valid if PSS available
sorted_cmds = cmds.items()
sorted_cmds.sort(lambda x, y:cmp(x[1], y[1]))
sorted_cmds = [x for x in sorted_cmds if x[1]]
return sorted_cmds, shareds, count, total
def print_header():
sys.stdout.write(" Private + Shared = RAM used\tProgram \n\n")
def print_memory_usage(sorted_cmds, shareds, count, total):
for cmd in sorted_cmds:
sys.stdout.write("%8sB + %8sB = %8sB\t%s\n" %
(human(cmd[1]-shareds[cmd[0]]),
human(shareds[cmd[0]]), human(cmd[1]),
cmd_with_count(cmd[0], count[cmd[0]])))
if have_pss:
sys.stdout.write("%s\n%s%8sB\n%s\n" %
("-" * 33, " " * 24, human(total), "=" * 33))
def verify_environment():
if os.geteuid() != 0:
sys.stderr.write("Sorry, root permission required.\n")
if __name__ == '__main__':
sys.stderr.close()
sys.exit(1)
try:
kv = kernel_ver()
except (IOError, OSError):
val = sys.exc_info()[1]
if val.errno == errno.ENOENT:
sys.stderr.write(
"Couldn't access /proc\n"
"Only GNU/Linux and FreeBSD (with linprocfs) are supported\n")
sys.exit(2)
else:
raise
if __name__ == '__main__':
verify_environment()
split_args, pids_to_show, watch = parse_options()
print_header()
if watch is not None:
try:
sorted_cmds = True
while sorted_cmds:
sorted_cmds, shareds, count, total = get_memory_usage( pids_to_show, split_args )
print_memory_usage(sorted_cmds, shareds, count, total)
time.sleep(watch)
else:
sys.stdout.write('Process does not exist anymore.\n')
except KeyboardInterrupt:
pass
else:
# This is the default behavior
sorted_cmds, shareds, count, total = get_memory_usage( pids_to_show, split_args )
print_memory_usage(sorted_cmds, shareds, count, total)
# We must close explicitly, so that any EPIPE exception
# is handled by our excepthook, rather than the default
# one which is reenabled after this script finishes.
sys.stdout.close()
vm_accuracy = shared_val_accuracy()
show_shared_val_accuracy( vm_accuracy )
Try to run it when you see the first instance of oom-killer, if really there is no RAM left than it's probably the bug. I'd double check though.
Remember to lsof -p on the RAM eater if you find it.
It may happen during peak hours.
Offline
The issue seems solved, but remains mysterious: After doing another full system upgrade this morning and rebooting, the issue did not reappear today although I was still on kernel 4.7.4-1 (4.7.5-1 was already listed as core here on the website this morning, but apparently there is a delay between what is listed here and the mirror I am using - the update has only reached me just now).
r0b0t, thanks for the script. It crashes when I try to run it, though - which version of Python does this require?
Offline
r0b0t's script is written in Python 2, so try running it with a Python 2 interpreter and see if it crashes.
Offline
From r0b0t's script
#!/usr/bin/env python
Yet
$ /usr/bin/env python
Python 3.5.2 (default, Jun 28 2016, 08:46:01)
[GCC 6.1.1 20160602] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
pacman -Qo /usr/bin/python
/usr/bin/python is owned by python 3.5.2-1
but
print help()
Is definately python2 not python3
Edit:
4.7.5-1 is now in core hopefully resolving the issue for you.
Last edited by loqs (2016-09-29 01:10:38)
Offline
Hi,
sorry forgot to point it out, it's python2 , you can check the top RAM eaters also with :
ps auxf | sort -nr -k 4 | head -100
ps -eo rss,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | egrep -v 0.00 | sort -n
You can find the original script here : https://github.com/pixelb/ps_mem/blob/master/ps_mem.py (props to pixelb)
Offline
I thought these issues have been fixed, but I keep getting random OOM kills while only 100KB of 16GB swap is used.
This happens both with completely stock kernel and linux-ck (from the repo-ck).
Here's the dmesg log from three days ago of two consecutive kills on the stock linux 4.8.2.1:
[88664.647172] Xorg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
[88664.647176] Xorg cpuset=/ mems_allowed=0
[88664.647180] CPU: 0 PID: 953 Comm: Xorg Tainted: P O 4.8.2-1-ARCH #1
[88664.647181] Hardware name: System manufacturer System Product Name/P8P67 REV 3.1, BIOS 3602 11/01/2012
[88664.647182] 0000000000000286 0000000011eed91e ffff8803f875b978 ffffffff812fe280
[88664.647184] ffff8803f875bb38 ffff880379bb3900 ffff8803f875b9e8 ffffffff81205f32
[88664.647185] ffff8803f875b980 0000000000000000 0000000000000000 0000000000000000
[88664.647186] Call Trace:
[88664.647191] [<ffffffff812fe280>] dump_stack+0x63/0x83
[88664.647194] [<ffffffff81205f32>] dump_header+0x5c/0x1ea
[88664.647195] [<ffffffff8117ca25>] oom_kill_process+0x265/0x410
[88664.647197] [<ffffffff810864c7>] ? has_capability_noaudit+0x17/0x20
[88664.647198] [<ffffffff8117cfa0>] out_of_memory+0x380/0x420
[88664.647201] [<ffffffff81313118>] ? find_next_bit+0x18/0x20
[88664.647202] [<ffffffff81182600>] __alloc_pages_nodemask+0xda0/0xde0
[88664.647205] [<ffffffff811d6585>] alloc_pages_current+0x95/0x140
[88664.647206] [<ffffffff811a3e3e>] kmalloc_order_trace+0x2e/0xf0
[88664.647207] [<ffffffff811e1eca>] __kmalloc+0x1ea/0x200
[88664.647218] [<ffffffffa17a02c7>] nvkms_alloc+0x27/0x60 [nvidia_modeset]
[88664.647222] [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88664.647228] [<ffffffffa17daf2a>] _nv001929kms+0x1a/0x30 [nvidia_modeset]
[88664.647234] [<ffffffffa17c7282>] ? _nv001878kms+0x32/0xcf0 [nvidia_modeset]
[88664.647235] [<ffffffff811e1eca>] ? __kmalloc+0x1ea/0x200
[88664.647236] [<ffffffff81205d54>] ? __check_object_size+0x54/0x1d6
[88664.647240] [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88664.647244] [<ffffffffa17a15ae>] ? _nv000169kms+0x2e/0x40 [nvidia_modeset]
[88664.647248] [<ffffffffa17a2101>] ? nvKmsIoctl+0x161/0x1e0 [nvidia_modeset]
[88664.647253] [<ffffffffa17a0da5>] ? nvkms_ioctl_common+0x45/0x80 [nvidia_modeset]
[88664.647257] [<ffffffffa17a0e51>] ? nvkms_ioctl+0x71/0xa0 [nvidia_modeset]
[88664.647391] [<ffffffffa0b7a080>] ? nvidia_frontend_compat_ioctl+0x40/0x50 [nvidia]
[88664.647431] [<ffffffffa0b7a09e>] ? nvidia_frontend_unlocked_ioctl+0xe/0x10 [nvidia]
[88664.647433] [<ffffffff8121c313>] ? do_vfs_ioctl+0xa3/0x5f0
[88664.647434] [<ffffffff81088f9b>] ? recalc_sigpending+0x1b/0x50
[88664.647435] [<ffffffff81089af1>] ? __set_task_blocked+0x41/0xa0
[88664.647436] [<ffffffff8121c8d9>] ? SyS_ioctl+0x79/0x90
[88664.647437] [<ffffffff8108c644>] ? SyS_rt_sigprocmask+0x74/0xc0
[88664.647439] [<ffffffff815f7cf2>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
[88664.647440] Mem-Info:
[88664.647443] active_anon:2772859 inactive_anon:277286 isolated_anon:0
active_file:453729 inactive_file:113612 isolated_file:0
unevictable:0 dirty:40975 writeback:0 unstable:0
slab_reclaimable:313689 slab_unreclaimable:24852
mapped:135832 shmem:70854 pagetables:33894 bounce:0
free:33374 free_pcp:144 free_cma:0
[88664.647445] Node 0 active_anon:11091436kB inactive_anon:1109144kB active_file:1814916kB inactive_file:454448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:543328kB dirty:163900kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 4982784kB anon_thp: 283416kB writeback_tmp:0kB unstable:0kB pages_scanned:55 all_unreclaimable? no
[88664.647446] Node 0 DMA free:15888kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88664.647448] lowmem_reserve[]: 0 3463 15989 15989 15989
[88664.647449] Node 0 DMA32 free:64616kB min:14620kB low:18272kB high:21924kB active_anon:2282848kB inactive_anon:56672kB active_file:579772kB inactive_file:170876kB unevictable:0kB writepending:18308kB present:3636768kB managed:3561020kB mlocked:0kB slab_reclaimable:333748kB slab_unreclaimable:18808kB kernel_stack:96kB pagetables:4372kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88664.647451] lowmem_reserve[]: 0 0 12526 12526 12526
[88664.647453] Node 0 Normal free:52992kB min:52892kB low:66112kB high:79332kB active_anon:8808588kB inactive_anon:1052472kB active_file:1235080kB inactive_file:283652kB unevictable:0kB writepending:145620kB present:13090816kB managed:12827296kB mlocked:0kB slab_reclaimable:921008kB slab_unreclaimable:80588kB kernel_stack:21472kB pagetables:131204kB bounce:0kB free_pcp:540kB local_pcp:0kB free_cma:0kB
[88664.647455] lowmem_reserve[]: 0 0 0 0 0
[88664.647456] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB
[88664.647462] Node 0 DMA32: 11181*4kB (UME) 2322*8kB (UME) 93*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64788kB
[88664.647466] Node 0 Normal: 3675*4kB (MEH) 4740*8kB (UMH) 0*16kB 3*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52844kB
[88664.647471] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[88664.647471] 638252 total pagecache pages
[88664.647472] 4 pages in swap cache
[88664.647473] Swap cache stats: add 94, delete 90, find 0/0
[88664.647473] Free swap = 16776844kB
[88664.647474] Total swap = 16777212kB
[88664.647474] 4185892 pages RAM
[88664.647474] 0 pages HighMem/MovableOnly
[88664.647474] 84838 pages reserved
[88664.647475] 0 pages hwpoisoned
[88664.647475] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[88664.647482] [ 357] 0 357 38691 7785 68 3 0 0 systemd-journal
[88664.647486] [ 465] 0 465 3032 539 11 3 0 0 paxd
[88664.647487] [ 474] 0 474 9141 947 19 3 0 -1000 systemd-udevd
[88664.647489] [ 647] 0 647 43505 360 18 3 0 0 lvmetad
[88664.647491] [ 780] 192 780 28539 970 26 3 0 0 systemd-timesyn
[88664.647492] [ 784] 84 784 10327 835 24 3 0 0 avahi-daemon
[88664.647493] [ 785] 84 785 10295 77 23 3 0 0 avahi-daemon
[88664.647494] [ 792] 0 792 3259 642 12 3 0 0 crond
[88664.647495] [ 793] 0 793 9629 1123 25 3 0 0 systemd-logind
[88664.647496] [ 796] 81 796 8355 995 21 3 0 -900 dbus-daemon
[88664.647497] [ 814] 0 814 108612 3521 74 4 0 0 NetworkManager
[88664.647498] [ 821] 1003 821 178158 6659 65 4 0 0 transmission-da
[88664.647499] [ 826] 0 826 21924 1682 47 3 0 0 cupsd
[88664.647500] [ 829] 0 829 10100 953 25 3 0 -1000 sshd
[88664.647500] [ 831] 0 831 22532 945 49 3 0 0 login
[88664.647501] [ 833] 124 833 76298 2522 51 4 0 0 colord
[88664.647502] [ 838] 0 838 64689 2320 60 4 0 0 cups-browsed
[88664.647503] [ 844] 102 844 131172 3821 53 4 0 0 polkitd
[88664.647504] [ 880] 0 880 3702 2182 12 3 0 0 dhclient
[88664.647505] [ 909] 1003 909 13724 1551 30 3 0 0 systemd
[88664.647506] [ 910] 1003 910 24712 461 49 3 0 0 (sd-pam)
[88664.647507] [ 917] 1003 917 3406 781 11 3 0 0 startx
[88664.647508] [ 952] 1003 952 3979 228 15 3 0 0 xinit
[88664.647510] [ 953] 1003 953 61019 14216 93 4 0 0 Xorg
[88664.647511] [ 963] 1003 963 32606 3724 66 4 0 0 i3
[88664.647513] [ 967] 1003 967 8219 738 21 3 0 0 dbus-daemon
[88664.647515] [ 973] 1003 973 15053 1103 32 3 0 0 redshift
[88664.647516] [ 979] 1003 979 5029 252 15 3 0 0 xautolock
[88664.647518] [ 985] 1003 985 28419 4532 59 4 0 0 compton
[88664.647519] [ 987] 1003 987 27077 1967 56 3 0 0 i3bar
[88664.647520] [ 990] 1003 990 292303 11908 65 5 0 0 urxvt
[88664.647522] [ 992] 1003 992 207220 22365 163 4 0 0 pcmanfm
[88664.647523] [ 993] 1003 993 189775 3626 45 4 0 0 python3
[88664.647524] [ 997] 1003 997 139110 3981 111 3 0 0 pulseaudio
[88664.647525] [ 1007] 1003 1007 18073 1449 29 3 0 0 xbindkeys
[88664.647526] [ 1008] 133 1008 44462 657 22 4 0 0 rtkit-daemon
[88664.647528] [ 1017] 1003 1017 69992 1508 36 4 0 0 gvfsd
[88664.647529] [ 1022] 1003 1022 103333 1345 36 4 0 0 gvfsd-fuse
[88664.647530] [ 1031] 1003 1031 4649 819 13 3 0 0 zsh
[88664.647531] [ 1035] 1003 1035 4887 714 14 3 0 0 tmux: client
[88664.647532] [ 1039] 1003 1039 6481 1830 17 3 0 0 tmux: server
[88664.647533] [ 1041] 1003 1041 3406 721 11 3 0 0 sh
[88664.647535] [ 1043] 1003 1043 9061 1699 23 3 0 0 zsh
[88664.647536] [ 1171] 1003 1171 166203 2876 62 4 0 0 gvfs-udisks2-vo
[88664.647537] [ 1217] 0 1217 116008 2325 52 4 0 0 udisksd
[88664.647538] [ 1371] 1003 1371 88997 1529 40 3 0 0 gvfsd-trash
[88664.647539] [ 1380] 1003 1380 89537 2046 41 4 0 0 gvfsd-computer
[88664.647540] [ 1387] 1003 1387 107970 1572 45 4 0 0 gvfsd-network
[88664.647541] [ 1393] 1003 1393 93102 1635 49 3 0 0 gvfsd-dnssd
[88664.647542] [ 1455] 1003 1455 49157 3350 94 3 0 0 vim
[88664.647543] [ 1710] 1003 1710 9779 1974 24 3 0 0 zsh
[88664.647544] [ 1742] 1003 1742 1039 17 8 3 0 0 sshtunnel
[88664.647545] [ 1744] 1003 1744 9646 1584 24 3 0 0 ssh
[88664.647546] [ 1807] 1003 1807 9779 1918 23 3 0 0 zsh
[88664.647547] [ 1904] 1003 1904 9518 1331 23 3 0 0 ssh
[88664.647548] [ 1932] 1003 1932 9568 707 22 3 0 0 ssh
[88664.647549] [ 1983] 99 1983 3406 684 13 3 0 0 pacserve
[88664.647550] [ 1984] 99 1984 90540 5776 78 4 0 0 python3
[88664.647550] [ 2013] 1003 2013 9518 1339 22 3 0 0 ssh
[88664.647551] [ 2017] 1003 2017 9546 625 21 3 0 0 ssh
[88664.647552] [ 2263] 1003 2263 10288 2328 24 3 0 0 zsh
[88664.647553] [ 2609] 1003 2609 475399 128767 602 6 0 0 chromium
[88664.647554] [ 2611] 1003 2611 1585 193 8 3 0 0 chrome-sandbox
[88664.647555] [ 2612] 1003 2612 115217 11933 170 4 0 0 chromium
[88664.647556] [ 2618] 1003 2618 1585 183 8 3 0 0 chrome-sandbox
[88664.647557] [ 2619] 1003 2619 7913 1108 22 4 0 0 nacl_helper
[88664.647558] [ 2623] 1003 2623 115217 2880 130 4 0 0 chromium
[88664.647559] [ 2706] 1003 2706 282557 128793 517 5 0 200 chromium
[88664.647560] [ 2712] 1003 2712 116433 6091 148 4 0 200 chromium
[88664.647561] [ 2720] 1003 2720 194399 9559 174 11 0 300 chromium
[88664.647562] [ 2726] 1003 2726 215159 17706 224 29 0 300 chromium
[88664.647563] [ 2742] 1003 2742 279001 63997 377 63 0 300 chromium
[88664.647564] [ 2750] 1003 2750 300872 76896 455 144 0 300 chromium
[88664.647565] [ 2754] 1003 2754 222911 24174 253 44 0 300 chromium
[88664.647566] [ 2762] 1003 2762 307503 99537 444 85 0 300 chromium
[88664.647567] [ 2771] 1003 2771 219930 20678 235 34 0 300 chromium
[88664.647567] [ 2835] 1003 2835 215415 18520 225 29 0 300 chromium
[88664.647569] [ 2839] 1003 2839 215415 17775 224 29 0 300 chromium
[88664.647570] [ 2866] 1003 2866 222592 21000 233 32 0 300 chromium
[88664.647571] [ 2886] 1003 2886 265074 43236 317 67 0 300 chromium
[88664.647571] [ 2896] 1003 2896 235280 31906 265 40 0 300 chromium
[88664.647572] [ 2899] 1003 2899 221167 22566 243 36 0 300 chromium
[88664.647573] [ 2905] 1003 2905 260572 49268 313 64 0 300 chromium
[88664.647574] [ 2911] 1003 2911 227028 28466 250 34 0 300 chromium
[88664.647575] [ 2928] 1003 2928 243407 33841 305 61 0 300 chromium
[88664.647577] [ 2932] 1003 2932 218563 20383 228 30 0 300 chromium
[88664.647578] [ 2943] 1003 2943 218816 21178 236 31 0 300 chromium
[88664.647579] [ 2947] 1003 2947 216752 20447 228 29 0 300 chromium
[88664.647580] [ 2954] 1003 2954 237878 29705 269 44 0 300 chromium
[88664.647581] [ 2958] 1003 2958 286159 77251 464 93 0 300 chromium
[88664.647582] [ 2987] 1003 2987 207988 17274 218 28 0 300 chromium
[88664.647583] [ 2993] 1003 2993 212086 17521 221 29 0 300 chromium
[88664.647583] [ 3004] 1003 3004 214646 19376 228 32 0 300 chromium
[88664.647584] [ 3008] 1003 3008 207988 18075 219 29 0 300 chromium
[88664.647585] [ 3021] 1003 3021 236335 30025 267 52 0 300 chromium
[88664.647587] [ 3042] 1003 3042 240257 34147 277 59 0 300 chromium
[88664.647588] [ 3133] 0 3133 75065 2029 45 3 0 0 upowerd
[88664.647589] [ 3250] 1003 3250 274553 68585 382 70 0 300 chromium
[88664.647591] [ 3292] 1003 3292 507638 185703 998 294 0 300 chromium
[88664.647592] [ 3328] 1003 3328 388671 164123 832 356 0 300 chromium
[88664.647593] [ 3415] 1003 3415 307696 88210 456 127 0 300 chromium
[88664.647595] [ 3536] 1003 3536 282332 78604 409 107 0 300 chromium
[88664.647596] [ 3546] 1003 3546 305499 101499 511 175 0 300 chromium
[88664.647598] [ 3674] 1003 3674 224280 25328 255 45 0 300 chromium
[88664.647599] [ 3694] 1003 3694 245342 43043 317 74 0 300 chromium
[88664.647600] [ 3747] 1003 3747 219705 24701 253 43 0 300 chromium
[88664.647601] [ 3756] 1003 3756 483482 240890 1059 340 0 300 chromium
[88664.647603] [ 3806] 1003 3806 360885 131138 629 210 0 300 chromium
[88664.647603] [ 3866] 1003 3866 283736 71387 428 120 0 300 chromium
[88664.647604] [ 3875] 1003 3875 424606 200196 769 242 0 300 chromium
[88664.647606] [ 3981] 1003 3981 372987 131164 673 223 0 300 chromium
[88664.647607] [ 4227] 1003 4227 230134 29402 269 51 0 300 chromium
[88664.647608] [ 4291] 1003 4291 288571 78957 435 90 0 300 chromium
[88664.647609] [ 4472] 1003 4472 282333 81069 413 107 0 300 chromium
[88664.647610] [ 4576] 1003 4576 277599 77367 405 116 0 300 chromium
[88664.647611] [ 4694] 1003 4694 232276 31002 274 49 0 300 chromium
[88664.647613] [ 4702] 1003 4702 256330 44934 335 82 0 300 chromium
[88664.647613] [ 4755] 1003 4755 262128 51582 351 83 0 300 chromium
[88664.647615] [ 4773] 1003 4773 222826 25658 254 43 0 300 chromium
[88664.647615] [ 4793] 1003 4793 245184 39982 310 75 0 300 chromium
[88664.647617] [ 4896] 1003 4896 237234 35088 286 59 0 300 chromium
[88664.647618] [ 4921] 1003 4921 236416 34098 287 61 0 300 chromium
[88664.647619] [ 4961] 1003 4961 239345 37760 293 62 0 300 chromium
[88664.647620] [ 4974] 1003 4974 244681 43793 305 60 0 300 chromium
[88664.647621] [ 5050] 1003 5050 231578 32100 273 49 0 300 chromium
[88664.647622] [ 5064] 1003 5064 247431 43004 313 71 0 300 chromium
[88664.647623] [ 5097] 1003 5097 238312 37053 296 64 0 300 chromium
[88664.647624] [ 5148] 1003 5148 254437 41322 339 84 0 300 chromium
[88664.647625] [ 5161] 1003 5161 237117 35520 293 59 0 300 chromium
[88664.647626] [ 5178] 1003 5178 221792 26662 259 49 0 300 chromium
[88664.647627] [ 5240] 1003 5240 223372 29933 265 50 0 300 chromium
[88664.647628] [ 5257] 1003 5257 222314 26071 261 48 0 300 chromium
[88664.647630] [ 5280] 1003 5280 222314 25925 258 48 0 300 chromium
[88664.647631] [ 5306] 1003 5306 240108 28375 273 44 0 300 chromium
[88664.647632] [ 5332] 1003 5332 222952 24615 257 44 0 300 chromium
[88664.647632] [ 5342] 1003 5342 229004 29209 265 46 0 300 chromium
[88664.647634] [ 5418] 1003 5418 232230 31798 276 51 0 300 chromium
[88664.647636] [ 5468] 1003 5468 215095 25277 254 48 0 300 chromium
[88664.647637] [ 5521] 1003 5521 222014 24540 252 44 0 300 chromium
[88664.647638] [ 5534] 1003 5534 216776 24237 253 48 0 300 chromium
[88664.647639] [ 5543] 1003 5543 306884 83475 496 124 0 300 chromium
[88664.647640] [ 5564] 1003 5564 239317 33578 297 62 0 300 chromium
[88664.647641] [ 5633] 1003 5633 246976 41563 330 84 0 300 chromium
[88664.647642] [ 5674] 1003 5674 250826 48486 329 73 0 300 chromium
[88664.647643] [ 5748] 1003 5748 244379 43492 303 60 0 300 chromium
[88664.647644] [ 5780] 1003 5780 316274 107199 514 163 0 300 chromium
[88664.647645] [ 5807] 1003 5807 267724 56806 365 98 0 300 chromium
[88664.647646] [ 5830] 1003 5830 279745 66927 404 107 0 300 chromium
[88664.647647] [ 5939] 1003 5939 247823 43061 326 73 0 300 chromium
[88664.647648] [ 7326] 1003 7326 164511 12264 203 15 0 300 chromium
[88664.647649] [23214] 1003 23214 243822 38613 292 49 0 300 chromium
[88664.647650] [20076] 1003 20076 86099 1942 36 3 0 0 at-spi-bus-laun
[88664.647652] [20081] 1003 20081 8186 723 21 3 0 0 dbus-daemon
[88664.647654] [20083] 1003 20083 55264 1449 42 3 0 0 at-spi2-registr
[88664.647656] [ 691] 1003 691 280810 49025 402 127 0 300 chromium
[88664.647658] [19755] 1003 19755 9779 1965 25 3 0 0 zsh
[88664.647659] [19924] 1003 19924 9779 1932 23 3 0 0 zsh
[88664.647660] [29338] 1003 29338 261044 44880 337 83 0 300 chromium
[88664.647661] [ 6482] 1003 6482 241143 29687 279 50 0 300 chromium
[88664.647663] [29081] 1003 29081 4649 829 13 3 0 0 zsh
[88664.647664] [29115] 1003 29115 242737 22002 163 5 0 0 Main
[88664.647665] [31563] 1003 31563 9485 1312 22 3 0 0 ssh
[88664.647666] [32492] 1003 32492 9485 1353 22 3 0 0 ssh
[88664.647667] [ 726] 1003 726 3983 53 13 3 0 0 xsel
[88664.647668] [ 3867] 1003 3867 495708 10455 61 5 0 0 urxvt
[88664.647669] [ 3868] 1003 3868 9778 1945 22 4 0 0 zsh
[88664.647670] [ 3970] 1003 3970 9485 1353 24 3 0 0 ssh
[88664.647672] [ 9907] 0 9907 17510 1306 37 3 0 0 sudo
[88664.647673] [ 9953] 0 9953 102926 14301 205 3 0 0 rsync
[88664.647675] [ 9954] 0 9954 39320 7677 82 3 0 0 rsync
[88664.647676] [ 9955] 0 9955 105278 9764 212 3 0 0 rsync
[88664.647677] Out of memory: Kill process 3756 (chromium) score 329 or sacrifice child
[88664.647701] Killed process 3756 (chromium) total-vm:1933928kB, anon-rss:888836kB, file-rss:73232kB, shmem-rss:1492kB
[88699.560672] Xorg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
[88699.560674] Xorg cpuset=/ mems_allowed=0
[88699.560678] CPU: 2 PID: 953 Comm: Xorg Tainted: P O 4.8.2-1-ARCH #1
[88699.560678] Hardware name: System manufacturer System Product Name/P8P67 REV 3.1, BIOS 3602 11/01/2012
[88699.560680] 0000000000000286 0000000011eed91e ffff8803f875b978 ffffffff812fe280
[88699.560681] ffff8803f875bb38 ffff880349c20e40 ffff8803f875b9e8 ffffffff81205f32
[88699.560683] ffff8803f875b980 0000000000000000 0000000000000000 0000000000000000
[88699.560684] Call Trace:
[88699.560690] [<ffffffff812fe280>] dump_stack+0x63/0x83
[88699.560692] [<ffffffff81205f32>] dump_header+0x5c/0x1ea
[88699.560694] [<ffffffff8117ca25>] oom_kill_process+0x265/0x410
[88699.560696] [<ffffffff810864c7>] ? has_capability_noaudit+0x17/0x20
[88699.560697] [<ffffffff8117cfa0>] out_of_memory+0x380/0x420
[88699.560699] [<ffffffff81313118>] ? find_next_bit+0x18/0x20
[88699.560701] [<ffffffff81182600>] __alloc_pages_nodemask+0xda0/0xde0
[88699.560704] [<ffffffff811d6585>] alloc_pages_current+0x95/0x140
[88699.560706] [<ffffffff811a3e3e>] kmalloc_order_trace+0x2e/0xf0
[88699.560707] [<ffffffff811e1eca>] __kmalloc+0x1ea/0x200
[88699.560720] [<ffffffffa17a02c7>] nvkms_alloc+0x27/0x60 [nvidia_modeset]
[88699.560725] [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88699.560731] [<ffffffffa17daf2a>] _nv001929kms+0x1a/0x30 [nvidia_modeset]
[88699.560737] [<ffffffffa17c7282>] ? _nv001878kms+0x32/0xcf0 [nvidia_modeset]
[88699.560738] [<ffffffff811e1eca>] ? __kmalloc+0x1ea/0x200
[88699.560739] [<ffffffff81205d54>] ? __check_object_size+0x54/0x1d6
[88699.560744] [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88699.560748] [<ffffffffa17a15ae>] ? _nv000169kms+0x2e/0x40 [nvidia_modeset]
[88699.560752] [<ffffffffa17a2101>] ? nvKmsIoctl+0x161/0x1e0 [nvidia_modeset]
[88699.560757] [<ffffffffa17a0da5>] ? nvkms_ioctl_common+0x45/0x80 [nvidia_modeset]
[88699.560761] [<ffffffffa17a0e51>] ? nvkms_ioctl+0x71/0xa0 [nvidia_modeset]
[88699.560907] [<ffffffffa0b7a080>] ? nvidia_frontend_compat_ioctl+0x40/0x50 [nvidia]
[88699.560945] [<ffffffffa0b7a09e>] ? nvidia_frontend_unlocked_ioctl+0xe/0x10 [nvidia]
[88699.560947] [<ffffffff8121c313>] ? do_vfs_ioctl+0xa3/0x5f0
[88699.560948] [<ffffffff81088f9b>] ? recalc_sigpending+0x1b/0x50
[88699.560949] [<ffffffff81089af1>] ? __set_task_blocked+0x41/0xa0
[88699.560950] [<ffffffff8121c8d9>] ? SyS_ioctl+0x79/0x90
[88699.560951] [<ffffffff8108c644>] ? SyS_rt_sigprocmask+0x74/0xc0
[88699.560954] [<ffffffff815f7cf2>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
[88699.560958] Mem-Info:
[88699.560960] active_anon:2556879 inactive_anon:277647 isolated_anon:0
active_file:496226 inactive_file:211590 isolated_file:64
unevictable:0 dirty:11020 writeback:10052 unstable:0
slab_reclaimable:385508 slab_unreclaimable:32365
mapped:139836 shmem:71185 pagetables:32910 bounce:0
free:33424 free_pcp:26 free_cma:0
[88699.560963] Node 0 active_anon:10227516kB inactive_anon:1110588kB active_file:1984904kB inactive_file:846360kB unevictable:0kB isolated(anon):0kB isolated(file):256kB mapped:559344kB dirty:44080kB writeback:40208kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 4503552kB anon_thp: 284740kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[88699.560964] Node 0 DMA free:15888kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88699.560966] lowmem_reserve[]: 0 3463 15989 15989 15989
[88699.560968] Node 0 DMA32 free:64652kB min:14620kB low:18272kB high:21924kB active_anon:2212312kB inactive_anon:56780kB active_file:610596kB inactive_file:132264kB unevictable:0kB writepending:29820kB present:3636768kB managed:3561020kB mlocked:0kB slab_reclaimable:444048kB slab_unreclaimable:14604kB kernel_stack:288kB pagetables:4196kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88699.560970] lowmem_reserve[]: 0 0 12526 12526 12526
[88699.560971] Node 0 Normal free:53156kB min:52892kB low:66112kB high:79332kB active_anon:8015204kB inactive_anon:1053808kB active_file:1374392kB inactive_file:714096kB unevictable:0kB writepending:54592kB present:13090816kB managed:12827296kB mlocked:0kB slab_reclaimable:1097984kB slab_unreclaimable:114844kB kernel_stack:21296kB pagetables:127444kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB
[88699.560973] lowmem_reserve[]: 0 0 0 0 0
[88699.560975] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB
[88699.560980] Node 0 DMA32: 5266*4kB (UME) 4744*8kB (UME) 353*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64664kB
[88699.560984] Node 0 Normal: 13104*4kB (UMH) 22*8kB (H) 0*16kB 11*32kB (H) 6*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 53328kB
[88699.560990] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[88699.560990] 779085 total pagecache pages
[88699.560991] 4 pages in swap cache
[88699.560992] Swap cache stats: add 94, delete 90, find 0/0
[88699.560992] Free swap = 16776844kB
[88699.560992] Total swap = 16777212kB
[88699.560993] 4185892 pages RAM
[88699.560993] 0 pages HighMem/MovableOnly
[88699.560993] 84838 pages reserved
[88699.560993] 0 pages hwpoisoned
[88699.560994] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[88699.561001] [ 357] 0 357 38691 8485 68 3 0 0 systemd-journal
[88699.561008] [ 465] 0 465 3032 539 11 3 0 0 paxd
[88699.561010] [ 474] 0 474 9141 947 19 3 0 -1000 systemd-udevd
[88699.561011] [ 647] 0 647 43505 360 18 3 0 0 lvmetad
[88699.561015] [ 780] 192 780 28539 970 26 3 0 0 systemd-timesyn
[88699.561016] [ 784] 84 784 10327 835 24 3 0 0 avahi-daemon
[88699.561017] [ 785] 84 785 10295 77 23 3 0 0 avahi-daemon
[88699.561018] [ 792] 0 792 3259 642 12 3 0 0 crond
[88699.561019] [ 793] 0 793 9629 1123 25 3 0 0 systemd-logind
[88699.561020] [ 796] 81 796 8355 995 21 3 0 -900 dbus-daemon
[88699.561021] [ 814] 0 814 108612 3521 74 4 0 0 NetworkManager
[88699.561022] [ 821] 1003 821 178158 6659 65 4 0 0 transmission-da
[88699.561023] [ 826] 0 826 21924 1682 47 3 0 0 cupsd
[88699.561025] [ 829] 0 829 10100 953 25 3 0 -1000 sshd
[88699.561026] [ 831] 0 831 22532 945 49 3 0 0 login
[88699.561027] [ 833] 124 833 76298 2522 51 4 0 0 colord
[88699.561028] [ 838] 0 838 64689 2320 60 4 0 0 cups-browsed
[88699.561029] [ 844] 102 844 131172 3821 53 4 0 0 polkitd
[88699.561030] [ 880] 0 880 3702 2182 12 3 0 0 dhclient
[88699.561031] [ 909] 1003 909 13724 1551 30 3 0 0 systemd
[88699.561032] [ 910] 1003 910 24712 461 49 3 0 0 (sd-pam)
[88699.561033] [ 917] 1003 917 3406 781 11 3 0 0 startx
[88699.561034] [ 952] 1003 952 3979 228 15 3 0 0 xinit
[88699.561035] [ 953] 1003 953 61019 14216 93 4 0 0 Xorg
[88699.561036] [ 963] 1003 963 32606 3724 66 4 0 0 i3
[88699.561038] [ 967] 1003 967 8219 738 21 3 0 0 dbus-daemon
[88699.561039] [ 973] 1003 973 15053 1103 32 3 0 0 redshift
[88699.561040] [ 979] 1003 979 5029 252 15 3 0 0 xautolock
[88699.561041] [ 985] 1003 985 28419 4532 59 4 0 0 compton
[88699.561042] [ 987] 1003 987 27077 1967 56 3 0 0 i3bar
[88699.561043] [ 990] 1003 990 292303 11908 65 5 0 0 urxvt
[88699.561044] [ 992] 1003 992 207220 22365 163 4 0 0 pcmanfm
[88699.561046] [ 993] 1003 993 189775 3626 45 4 0 0 python3
[88699.561047] [ 997] 1003 997 139110 3981 111 3 0 0 pulseaudio
[88699.561048] [ 1007] 1003 1007 18073 1449 29 3 0 0 xbindkeys
[88699.561049] [ 1008] 133 1008 44462 657 22 4 0 0 rtkit-daemon
[88699.561050] [ 1017] 1003 1017 69992 1508 36 4 0 0 gvfsd
[88699.561051] [ 1022] 1003 1022 103333 1345 36 4 0 0 gvfsd-fuse
[88699.561052] [ 1031] 1003 1031 4649 819 13 3 0 0 zsh
[88699.561054] [ 1035] 1003 1035 4887 714 14 3 0 0 tmux: client
[88699.561054] [ 1039] 1003 1039 7035 2369 18 3 0 0 tmux: server
[88699.561055] [ 1041] 1003 1041 3406 721 11 3 0 0 sh
[88699.561056] [ 1043] 1003 1043 9061 1699 23 3 0 0 zsh
[88699.561058] [ 1171] 1003 1171 166203 2876 62 4 0 0 gvfs-udisks2-vo
[88699.561058] [ 1217] 0 1217 116008 2325 52 4 0 0 udisksd
[88699.561059] [ 1371] 1003 1371 88997 1529 40 3 0 0 gvfsd-trash
[88699.561061] [ 1380] 1003 1380 89537 2046 41 4 0 0 gvfsd-computer
[88699.561062] [ 1387] 1003 1387 107970 1572 45 4 0 0 gvfsd-network
[88699.561063] [ 1393] 1003 1393 93102 1635 49 3 0 0 gvfsd-dnssd
[88699.561064] [ 1455] 1003 1455 49157 3350 94 3 0 0 vim
[88699.561066] [ 1710] 1003 1710 9779 1974 24 3 0 0 zsh
[88699.561068] [ 1742] 1003 1742 1039 17 8 3 0 0 sshtunnel
[88699.561070] [ 1744] 1003 1744 9646 1584 24 3 0 0 ssh
[88699.561072] [ 1807] 1003 1807 9779 1918 23 3 0 0 zsh
[88699.561073] [ 1904] 1003 1904 9518 1331 23 3 0 0 ssh
[88699.561075] [ 1932] 1003 1932 9568 707 22 3 0 0 ssh
[88699.561077] [ 1983] 99 1983 3406 684 13 3 0 0 pacserve
[88699.561078] [ 1984] 99 1984 90540 5776 78 4 0 0 python3
[88699.561080] [ 2013] 1003 2013 9518 1339 22 3 0 0 ssh
[88699.561082] [ 2017] 1003 2017 9546 625 21 3 0 0 ssh
[88699.561084] [ 2263] 1003 2263 10288 2328 24 3 0 0 zsh
[88699.561085] [ 2609] 1003 2609 475468 131371 602 6 0 0 chromium
[88699.561086] [ 2611] 1003 2611 1585 193 8 3 0 0 chrome-sandbox
[88699.561089] [ 2612] 1003 2612 115217 11933 170 4 0 0 chromium
[88699.561090] [ 2618] 1003 2618 1585 183 8 3 0 0 chrome-sandbox
[88699.561092] [ 2619] 1003 2619 7913 1108 22 4 0 0 nacl_helper
[88699.561093] [ 2623] 1003 2623 115217 2880 130 4 0 0 chromium
[88699.561096] [ 2706] 1003 2706 281580 129119 517 5 0 200 chromium
[88699.561097] [ 2712] 1003 2712 116433 6091 148 4 0 200 chromium
[88699.561099] [ 2720] 1003 2720 194399 9303 174 11 0 300 chromium
[88699.561102] [ 2726] 1003 2726 215159 17706 224 29 0 300 chromium
[88699.561104] [ 2742] 1003 2742 279001 64038 377 63 0 300 chromium
[88699.561106] [ 2750] 1003 2750 300123 75558 450 139 0 300 chromium
[88699.561107] [ 2754] 1003 2754 222911 24174 253 44 0 300 chromium
[88699.561108] [ 2762] 1003 2762 307503 99537 444 85 0 300 chromium
[88699.561110] [ 2771] 1003 2771 219930 20678 235 34 0 300 chromium
[88699.561112] [ 2835] 1003 2835 215415 18520 225 29 0 300 chromium
[88699.561114] [ 2839] 1003 2839 215415 17775 224 29 0 300 chromium
[88699.561115] [ 2866] 1003 2866 222592 21000 233 32 0 300 chromium
[88699.561117] [ 2886] 1003 2886 265074 43120 317 67 0 300 chromium
[88699.561118] [ 2896] 1003 2896 235280 31856 265 40 0 300 chromium
[88699.561120] [ 2899] 1003 2899 221167 22566 243 36 0 300 chromium
[88699.561122] [ 2905] 1003 2905 260572 49416 313 64 0 300 chromium
[88699.561124] [ 2911] 1003 2911 227028 28466 250 34 0 300 chromium
[88699.561125] [ 2928] 1003 2928 243407 33841 305 61 0 300 chromium
[88699.561127] [ 2932] 1003 2932 218563 20449 228 30 0 300 chromium
[88699.561129] [ 2943] 1003 2943 218816 21178 236 31 0 300 chromium
[88699.561131] [ 2947] 1003 2947 216752 20447 228 29 0 300 chromium
[88699.561133] [ 2954] 1003 2954 237878 29705 269 44 0 300 chromium
[88699.561135] [ 2958] 1003 2958 286159 77257 465 94 0 300 chromium
[88699.561136] [ 2987] 1003 2987 207988 17274 218 28 0 300 chromium
[88699.561138] [ 2993] 1003 2993 212086 17521 221 29 0 300 chromium
[88699.561140] [ 3004] 1003 3004 214646 19376 228 32 0 300 chromium
[88699.561142] [ 3008] 1003 3008 207988 18075 219 29 0 300 chromium
[88699.561144] [ 3021] 1003 3021 236335 30046 267 52 0 300 chromium
[88699.561146] [ 3042] 1003 3042 240257 34080 277 59 0 300 chromium
[88699.561147] [ 3133] 0 3133 75065 2029 45 3 0 0 upowerd
[88699.561148] [ 3250] 1003 3250 274553 68585 382 70 0 300 chromium
[88699.561150] [ 3292] 1003 3292 508150 186089 1001 296 0 300 chromium
[88699.561151] [ 3328] 1003 3328 388671 164080 832 356 0 300 chromium
[88699.561153] [ 3415] 1003 3415 307696 88364 456 127 0 300 chromium
[88699.561155] [ 3536] 1003 3536 282332 78604 409 107 0 300 chromium
[88699.561157] [ 3546] 1003 3546 305499 101551 511 175 0 300 chromium
[88699.561159] [ 3674] 1003 3674 224280 25328 255 45 0 300 chromium
[88699.561161] [ 3694] 1003 3694 245342 43043 317 74 0 300 chromium
[88699.561163] [ 3747] 1003 3747 219705 24701 253 43 0 300 chromium
[88699.561165] [ 3806] 1003 3806 360885 131169 629 210 0 300 chromium
[88699.561167] [ 3866] 1003 3866 283736 71387 428 120 0 300 chromium
[88699.561169] [ 3875] 1003 3875 499231 201349 782 252 0 300 chromium
[88699.561170] [ 3981] 1003 3981 372987 131031 673 223 0 300 chromium
[88699.561173] [ 4227] 1003 4227 230134 29402 269 51 0 300 chromium
[88699.561175] [ 4291] 1003 4291 288571 79023 435 90 0 300 chromium
[88699.561177] [ 4472] 1003 4472 282333 81144 413 107 0 300 chromium
[88699.561179] [ 4576] 1003 4576 277599 77367 405 116 0 300 chromium
[88699.561181] [ 4694] 1003 4694 232276 31002 274 49 0 300 chromium
[88699.561183] [ 4702] 1003 4702 256330 45063 335 82 0 300 chromium
[88699.561185] [ 4755] 1003 4755 262128 51582 351 83 0 300 chromium
[88699.561186] [ 4773] 1003 4773 222826 25658 254 43 0 300 chromium
[88699.561189] [ 4793] 1003 4793 245184 39989 310 75 0 300 chromium
[88699.561191] [ 4896] 1003 4896 237234 35088 286 59 0 300 chromium
[88699.561194] [ 4921] 1003 4921 236416 34098 287 61 0 300 chromium
[88699.561196] [ 4961] 1003 4961 239345 37760 293 62 0 300 chromium
[88699.561197] [ 4974] 1003 4974 244681 43793 305 60 0 300 chromium
[88699.561199] [ 5050] 1003 5050 231578 32100 273 49 0 300 chromium
[88699.561200] [ 5064] 1003 5064 247431 43004 313 71 0 300 chromium
[88699.561202] [ 5097] 1003 5097 238312 37053 296 64 0 300 chromium
[88699.561204] [ 5148] 1003 5148 254437 41197 339 84 0 300 chromium
[88699.561206] [ 5161] 1003 5161 237117 35520 293 59 0 300 chromium
[88699.561208] [ 5178] 1003 5178 221792 26662 259 49 0 300 chromium
[88699.561210] [ 5240] 1003 5240 223372 29933 265 50 0 300 chromium
[88699.561212] [ 5257] 1003 5257 222314 26071 261 48 0 300 chromium
[88699.561214] [ 5280] 1003 5280 222314 25925 258 48 0 300 chromium
[88699.561217] [ 5306] 1003 5306 240108 28375 273 44 0 300 chromium
[88699.561219] [ 5332] 1003 5332 222952 24615 257 44 0 300 chromium
[88699.561221] [ 5342] 1003 5342 229004 29209 265 46 0 300 chromium
[88699.561224] [ 5418] 1003 5418 232230 31798 276 51 0 300 chromium
[88699.561226] [ 5468] 1003 5468 215095 25277 254 48 0 300 chromium
[88699.561228] [ 5521] 1003 5521 222014 24540 252 44 0 300 chromium
[88699.561230] [ 5534] 1003 5534 216776 24237 253 48 0 300 chromium
[88699.561232] [ 5543] 1003 5543 306884 83528 496 124 0 300 chromium
[88699.561234] [ 5564] 1003 5564 239317 33383 296 61 0 300 chromium
[88699.561237] [ 5633] 1003 5633 246976 41634 330 84 0 300 chromium
[88699.561239] [ 5674] 1003 5674 250826 48486 329 73 0 300 chromium
[88699.561242] [ 5748] 1003 5748 244379 43492 303 60 0 300 chromium
[88699.561245] [ 5780] 1003 5780 316274 107265 514 163 0 300 chromium
[88699.561247] [ 5807] 1003 5807 267686 56227 363 96 0 300 chromium
[88699.561249] [ 5830] 1003 5830 279745 66861 404 107 0 300 chromium
[88699.561251] [ 5939] 1003 5939 247823 43080 326 73 0 300 chromium
[88699.561253] [ 7326] 1003 7326 164511 12264 203 15 0 300 chromium
[88699.561255] [23214] 1003 23214 243822 38613 292 49 0 300 chromium
[88699.561257] [20076] 1003 20076 86099 1942 36 3 0 0 at-spi-bus-laun
[88699.561259] [20081] 1003 20081 8186 723 21 3 0 0 dbus-daemon
[88699.561261] [20083] 1003 20083 55264 1449 42 3 0 0 at-spi2-registr
[88699.561263] [ 691] 1003 691 280810 48909 402 127 0 300 chromium
[88699.561266] [19755] 1003 19755 9779 1965 25 3 0 0 zsh
[88699.561270] [19924] 1003 19924 9779 1932 23 3 0 0 zsh
[88699.561272] [29338] 1003 29338 261044 44978 337 83 0 300 chromium
[88699.561275] [ 6482] 1003 6482 241143 29687 279 50 0 300 chromium
[88699.561280] [29081] 1003 29081 4649 829 13 3 0 0 zsh
[88699.561282] [29115] 1003 29115 242737 22002 163 5 0 0 Main
[88699.561285] [31563] 1003 31563 9485 1312 22 3 0 0 ssh
[88699.561286] [32492] 1003 32492 9485 1353 22 3 0 0 ssh
[88699.561288] [ 726] 1003 726 3983 53 13 3 0 0 xsel
[88699.561290] [ 3867] 1003 3867 495708 10455 61 5 0 0 urxvt
[88699.561292] [ 3868] 1003 3868 9778 1945 22 4 0 0 zsh
[88699.561295] [ 3970] 1003 3970 9485 1353 24 3 0 0 ssh
[88699.561298] [ 9907] 0 9907 17510 1306 37 3 0 0 sudo
[88699.561301] [ 9953] 0 9953 103054 16330 206 3 0 0 rsync
[88699.561303] [ 9954] 0 9954 39448 8359 82 3 0 0 rsync
[88699.561304] [ 9955] 0 9955 105406 11564 213 3 0 0 rsync
[88699.561306] Out of memory: Kill process 3875 (chromium) score 324 or sacrifice child
[88699.561352] Killed process 3875 (chromium) total-vm:1996924kB, anon-rss:702236kB, file-rss:73416kB, shmem-rss:29744kB
And here it is happening just now on linux-ck 4.8.4.1:
[82892.878181] Xorg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
[82892.878183] Xorg cpuset=/ mems_allowed=0
[82892.878188] CPU: 2 PID: 862 Comm: Xorg Tainted: P O 4.8.4-1-ck #1
[82892.878189] Hardware name: System manufacturer System Product Name/P8P67 REV 3.1, BIOS 3602 11/01/2012
[82892.878190] 0000000000000286 0000000053b3df50 ffff8803f6adf978 ffffffff812f2af0
[82892.878192] ffff8803f6adfb38 ffff88032c188000 ffff8803f6adf9e8 ffffffff811e6a91
[82892.878193] ffffffff810a94e7 ffffffff8184c700 0000000000000246 ffff8803f6adf9c0
[82892.878194] Call Trace:
[82892.878201] [<ffffffff812f2af0>] dump_stack+0x63/0x83
[82892.878203] [<ffffffff811e6a91>] dump_header+0x5c/0x1ea
[82892.878205] [<ffffffff810a94e7>] ? swake_up+0x37/0x40
[82892.878207] [<ffffffff810c586d>] ? rcu_report_qs_rsp+0x4d/0x80
[82892.878208] [<ffffffff8115f035>] oom_kill_process+0x265/0x410
[82892.878209] [<ffffffff810ca214>] ? rcu_read_unlock_special+0x294/0x2d0
[82892.878210] [<ffffffff8115f5b0>] out_of_memory+0x380/0x420
[82892.878211] [<ffffffff813077f8>] ? find_next_bit+0x18/0x20
[82892.878213] [<ffffffff81164b90>] __alloc_pages_nodemask+0xda0/0xde0
[82892.878215] [<ffffffff811b8295>] alloc_pages_current+0x95/0x140
[82892.878217] [<ffffffff811860fe>] kmalloc_order_trace+0x2e/0xf0
[82892.878218] [<ffffffff811c3a1a>] __kmalloc+0x1ea/0x200
[82892.878232] [<ffffffffa078e297>] nvkms_alloc+0x27/0x60 [nvidia_modeset]
[82892.878237] [<ffffffffa078f540>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[82892.878243] [<ffffffffa07c8eea>] _nv001929kms+0x1a/0x30 [nvidia_modeset]
[82892.878249] [<ffffffffa07b5242>] ? _nv001878kms+0x32/0xcf0 [nvidia_modeset]
[82892.878250] [<ffffffff811c3a1a>] ? __kmalloc+0x1ea/0x200
[82892.878251] [<ffffffff811e68b4>] ? __check_object_size+0x54/0x1d5
[82892.878255] [<ffffffffa078f540>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[82892.878260] [<ffffffffa078f56e>] ? _nv000169kms+0x2e/0x40 [nvidia_modeset]
[82892.878264] [<ffffffffa07900c1>] ? nvKmsIoctl+0x161/0x1e0 [nvidia_modeset]
[82892.878268] [<ffffffffa078ed65>] ? nvkms_ioctl_common+0x45/0x80 [nvidia_modeset]
[82892.878273] [<ffffffffa078ee11>] ? nvkms_ioctl+0x71/0xa0 [nvidia_modeset]
[82892.878358] [<ffffffffa08f9080>] ? nvidia_frontend_compat_ioctl+0x40/0x50 [nvidia]
[82892.878396] [<ffffffffa08f909e>] ? nvidia_frontend_unlocked_ioctl+0xe/0x10 [nvidia]
[82892.878397] [<ffffffff811fce33>] ? do_vfs_ioctl+0xa3/0x5f0
[82892.878398] [<ffffffff811fd3f9>] ? SyS_ioctl+0x79/0x90
[82892.878400] [<ffffffff815eb8b2>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
[82892.878411] Mem-Info:
[82892.878414] active_anon:2231054 inactive_anon:273281 isolated_anon:0
active_file:838912 inactive_file:189143 isolated_file:34
unevictable:0 dirty:2959 writeback:12020 unstable:0
slab_reclaimable:42926 slab_unreclaimable:55448
mapped:159897 shmem:110433 pagetables:31901 bounce:0
free:33428 free_pcp:87 free_cma:0
[82892.878417] Node 0 active_anon:8924216kB inactive_anon:1093124kB active_file:3355648kB inactive_file:756572kB unevictable:0kB isolated(anon):0kB isolated(file):136kB mapped:639588kB dirty:11836kB writeback:48080kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 2617344kB anon_thp: 441732kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[82892.878417] Node 0 DMA free:15888kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[82892.878421] lowmem_reserve[]: 0 3463 15989 15989 15989
[82892.878422] Node 0 DMA32 free:64536kB min:14620kB low:18272kB high:21924kB active_anon:1834916kB inactive_anon:69556kB active_file:735752kB inactive_file:119064kB unevictable:0kB writepending:11896kB present:3636768kB managed:3561072kB mlocked:0kB slab_reclaimable:48824kB slab_unreclaimable:68748kB kernel_stack:992kB pagetables:5756kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[82892.878425] lowmem_reserve[]: 0 0 12526 12526 12526
[82892.878426] Node 0 Normal free:53288kB min:52892kB low:66112kB high:79332kB active_anon:7089300kB inactive_anon:1023568kB active_file:2619896kB inactive_file:637636kB unevictable:0kB writepending:48024kB present:13090816kB managed:12827312kB mlocked:0kB slab_reclaimable:122880kB slab_unreclaimable:153228kB kernel_stack:31584kB pagetables:121848kB bounce:0kB free_pcp:300kB local_pcp:120kB free_cma:0kB
[82892.878428] lowmem_reserve[]: 0 0 0 0 0
[82892.878429] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB
[82892.878435] Node 0 DMA32: 4259*4kB (UME) 1377*8kB (UME) 2141*16kB (UMEH) 15*32kB (UH) 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 64772kB
[82892.878440] Node 0 Normal: 8298*4kB (MEH) 2004*8kB (UMEH) 141*16kB (UMEH) 23*32kB (UMH) 6*64kB (MH) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52856kB
[82892.878446] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[82892.878446] 1138511 total pagecache pages
[82892.878447] 0 pages in swap cache
[82892.878448] Swap cache stats: add 29, delete 29, find 0/0
[82892.878448] Free swap = 16777096kB
[82892.878449] Total swap = 16777212kB
[82892.878449] 4185892 pages RAM
[82892.878449] 0 pages HighMem/MovableOnly
[82892.878450] 84821 pages reserved
[82892.878450] 0 pages hwpoisoned
[82892.878450] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[82892.878459] [ 367] 0 367 35484 5074 65 3 0 0 systemd-journal
[82892.878461] [ 384] 0 384 3032 478 10 3 0 0 paxd
[82892.878463] [ 396] 0 396 9183 873 19 3 0 -1000 systemd-udevd
[82892.878465] [ 561] 0 561 43505 267 19 3 0 0 lvmetad
[82892.878469] [ 690] 192 690 28539 940 26 3 0 0 systemd-timesyn
[82892.878470] [ 695] 0 695 9629 1086 21 3 0 0 systemd-logind
[82892.878471] [ 696] 81 696 8364 925 21 3 0 -900 dbus-daemon
[82892.878472] [ 703] 0 703 3259 652 11 3 0 0 crond
[82892.878474] [ 704] 84 704 10327 720 26 3 0 0 avahi-daemon
[82892.878475] [ 705] 84 705 10295 76 24 3 0 0 avahi-daemon
[82892.878476] [ 711] 0 711 90175 2130 75 4 0 0 NetworkManager
[82892.878477] [ 728] 0 728 21924 1242 45 4 0 0 cupsd
[82892.878478] [ 731] 0 731 10100 970 24 3 0 -1000 sshd
[82892.878480] [ 734] 1003 734 178158 5376 63 4 0 0 transmission-da
[82892.878481] [ 737] 124 737 76308 2026 49 3 0 0 colord
[82892.878482] [ 742] 0 742 22554 875 47 3 0 0 login
[82892.878484] [ 745] 0 745 64689 1503 59 3 0 0 cups-browsed
[82892.878485] [ 750] 102 750 131172 2872 51 3 0 0 polkitd
[82892.878487] [ 806] 0 806 3702 2110 11 3 0 0 dhclient
[82892.878488] [ 818] 1003 818 13726 1364 31 3 0 0 systemd
[82892.878489] [ 820] 1003 820 24713 457 49 3 0 0 (sd-pam)
[82892.878490] [ 826] 1003 826 3406 782 11 3 0 0 startx
[82892.878492] [ 861] 1003 861 3979 218 14 3 0 0 xinit
[82892.878493] [ 862] 1003 862 72339 26684 118 3 0 0 Xorg
[82892.878494] [ 870] 1003 870 33095 2069 67 3 0 0 i3
[82892.878495] [ 874] 1003 874 8246 828 20 3 0 0 dbus-daemon
[82892.878496] [ 880] 1003 880 15053 884 32 3 0 0 redshift
[82892.878497] [ 886] 1003 886 5029 252 15 3 0 0 xautolock
[82892.878499] [ 892] 1003 892 28428 4053 58 4 0 0 compton
[82892.878500] [ 899] 1003 899 191670 25064 168 4 0 0 pcmanfm
[82892.878501] [ 904] 1003 904 171855 6585 115 4 0 0 pulseaudio
[82892.878503] [ 914] 1003 914 18073 1382 27 3 0 0 xbindkeys
[82892.878504] [ 915] 133 915 44462 679 22 4 0 0 rtkit-daemon
[82892.878506] [ 924] 1003 924 69992 995 36 3 0 0 gvfsd
[82892.878507] [ 929] 1003 929 103333 1409 35 3 0 0 gvfsd-fuse
[82892.878508] [ 1068] 1003 1068 166224 1670 60 4 0 0 gvfs-udisks2-vo
[82892.878510] [ 1095] 0 1095 116030 1900 51 4 0 0 udisksd
[82892.878511] [ 1237] 1003 1237 1039 17 8 3 0 0 sshtunnel
[82892.878513] [ 1240] 1003 1240 9738 1646 23 3 0 0 ssh
[82892.878514] [ 1286] 1003 1286 89015 1482 41 3 0 0 gvfsd-trash
[82892.878515] [ 1295] 1003 1295 89537 1019 41 3 0 0 gvfsd-computer
[82892.878516] [ 1301] 1003 1301 107970 1073 43 3 0 0 gvfsd-network
[82892.878518] [ 1307] 1003 1307 93101 1170 47 4 0 0 gvfsd-dnssd
[82892.878519] [ 1469] 99 1469 3406 715 13 3 0 0 pacserve
[82892.878521] [ 1470] 99 1470 90555 4746 77 4 0 0 python3
[82892.878522] [ 1689] 1003 1689 9553 658 22 3 0 0 ssh
[82892.878523] [ 1952] 1003 1952 9543 684 23 3 0 0 ssh
[82892.878525] [ 2986] 1003 2986 1021587 162893 772 7 0 0 chromium
[82892.878526] [ 2996] 1003 2996 1585 185 9 3 0 0 chrome-sandbox
[82892.878527] [ 2997] 1003 2997 115614 4693 174 3 0 0 chromium
[82892.878529] [ 3004] 1003 3004 115614 2544 126 3 0 0 chromium
[82892.878530] [ 3094] 1003 3094 290565 76174 380 5 0 200 chromium
[82892.878532] [ 3098] 1003 3098 353034 19468 257 41 0 300 chromium
[82892.878533] [ 3104] 1003 3104 341767 12691 224 28 0 300 chromium
[82892.878534] [ 3107] 1003 3107 350032 16478 236 34 0 300 chromium
[82892.878536] [ 3110] 1003 3110 357266 20262 393 46 0 300 chromium
[82892.878540] [ 3113] 1003 3113 392215 49743 382 49 0 300 chromium
[82892.878541] [ 3116] 1003 3116 346612 14140 228 31 0 300 chromium
[82892.878542] [ 3119] 1003 3119 356344 16964 237 32 0 300 chromium
[82892.878544] [ 3130] 1003 3130 387860 39041 317 70 0 300 chromium
[82892.878545] [ 3135] 1003 3135 366765 27022 275 42 0 300 chromium
[82892.878546] [ 3139] 1003 3139 372628 46238 312 58 0 300 chromium
[82892.878547] [ 3143] 1003 3143 363473 25922 275 42 0 300 chromium
[82892.878549] [ 3147] 1003 3147 384348 42765 388 130 0 300 chromium
[82892.878550] [ 3151] 1003 3151 351902 15572 236 31 0 300 chromium
[82892.878551] [ 3155] 1003 3155 406886 43203 401 129 0 300 chromium
[82892.878552] [ 3158] 1003 3158 355231 22323 258 39 0 300 chromium
[82892.878553] [ 3163] 1003 3163 348459 16402 235 31 0 300 chromium
[82892.878554] [ 3166] 1003 3166 372618 26599 280 47 0 300 chromium
[82892.878555] [ 3171] 1003 3171 310541 6398 180 12 0 300 chromium
[82892.878557] [ 3175] 1003 3175 320742 12958 222 29 0 300 chromium
[82892.878557] [ 3178] 1003 3178 339175 12650 222 28 0 300 chromium
[82892.878558] [ 3182] 1003 3182 347881 14879 233 32 0 300 chromium
[82892.878559] [ 3187] 1003 3187 341264 12989 224 28 0 300 chromium
[82892.878561] [ 3195] 1003 3195 375319 31495 303 58 0 300 chromium
[82892.878562] [ 3199] 1003 3199 371535 34963 296 76 0 300 chromium
[82892.878563] [ 3203] 1003 3203 370441 24485 267 41 0 300 chromium
[82892.878564] [ 3212] 1003 3212 132571 4960 140 3 0 200 chromium
[82892.878565] [ 3265] 1003 3265 460787 92492 628 200 0 300 chromium
[82892.878566] [ 3269] 1003 3269 368165 34058 319 78 0 300 chromium
[82892.878567] [ 3552] 0 3552 75043 1428 45 4 0 0 upowerd
[82892.878569] [ 4027] 1003 4027 390763 36076 395 61 0 300 chromium
[82892.878570] [ 4157] 1003 4157 382716 33568 352 59 0 300 chromium
[82892.878571] [ 4169] 1003 4169 386368 46484 369 103 0 300 chromium
[82892.878572] [ 4461] 1003 4461 389677 56117 392 84 0 300 chromium
[82892.878573] [ 4855] 1003 4855 373208 33335 307 54 0 300 chromium
[82892.878574] [ 4868] 1003 4868 387365 45880 367 78 0 300 chromium
[82892.878575] [ 4880] 1003 4880 371951 29088 316 51 0 300 chromium
[82892.878577] [ 4945] 1003 4945 478921 121459 643 204 0 300 chromium
[82892.878578] [ 4964] 1003 4964 370792 29137 304 55 0 300 chromium
[82892.878579] [ 5121] 1003 5121 389788 34942 322 65 0 300 chromium
[82892.878581] [ 5537] 1003 5537 381149 30029 310 63 0 300 chromium
[82892.878583] [ 5705] 1003 5705 365445 26076 303 49 0 300 chromium
[82892.878584] [ 5714] 1003 5714 379106 43542 382 95 0 300 chromium
[82892.878585] [ 5804] 1003 5804 394113 38354 412 57 0 300 chromium
[82892.878586] [ 5819] 1003 5819 375900 31106 308 66 0 300 chromium
[82892.878587] [ 5836] 1003 5836 486490 143378 629 103 0 300 chromium
[82892.878589] [ 5849] 1003 5849 379252 32987 314 73 0 300 chromium
[82892.878590] [ 5873] 1003 5873 386554 42809 413 100 0 300 chromium
[82892.878591] [ 5900] 1003 5900 347168 17964 248 44 0 300 chromium
[82892.878592] [ 5927] 1003 5927 359704 22244 268 43 0 300 chromium
[82892.878593] [ 6013] 1003 6013 375234 36920 327 69 0 300 chromium
[82892.878594] [ 7842] 1003 7842 49039 1330 32 3 0 0 gvfsd-metadata
[82892.878596] [30964] 1003 30964 27077 1272 56 3 0 0 i3bar
[82892.878597] [30965] 1003 30965 370002 3063 59 4 0 0 python3
[82892.878598] [12646] 1003 12646 86099 1048 36 4 0 0 at-spi-bus-laun
[82892.878600] [12651] 1003 12651 8186 645 20 3 0 0 dbus-daemon
[82892.878601] [12653] 1003 12653 55297 1084 42 3 0 0 at-spi2-registr
[82892.878602] [25052] 1003 25052 16400 1056 36 3 0 0 gconfd-2
[82892.878603] [ 1137] 1003 1137 291473 9809 65 5 0 0 urxvt
[82892.878605] [ 1147] 1003 1147 9157 1481 23 3 0 0 zsh
[82892.878606] [ 1201] 1003 1201 4887 624 14 3 0 0 tmux: client
[82892.878607] [ 1203] 1003 1203 12229 7235 29 3 0 0 tmux: server
[82892.878608] [ 1204] 1003 1204 3406 720 11 3 0 0 sh
[82892.878609] [ 1577] 1003 1577 531498 5694 156 6 0 0 vim
[82892.878610] [ 2617] 1003 2617 9778 1612 23 3 0 0 zsh
[82892.878611] [ 2712] 1003 2712 9778 1587 24 3 0 0 zsh
[82892.878613] [ 2809] 1003 2809 9485 1317 25 4 0 0 ssh
[82892.878614] [ 2873] 1003 2873 9485 1295 23 3 0 0 ssh
[82892.878615] [11159] 1003 11159 9793 1676 23 3 0 0 zsh
[82892.878616] [12982] 1003 12982 9957 1823 23 3 0 0 zsh
[82892.878617] [13082] 1003 13082 9158 1473 23 3 0 0 zsh
[82892.878619] [13815] 1003 13815 4093 1380 13 3 0 0 htop
[82892.878620] [28960] 1003 28960 496894 158843 782 195 0 300 chromium
[82892.878621] [28964] 1003 28964 461217 109491 660 283 0 300 chromium
[82892.878622] [28972] 1003 28972 384940 31482 312 65 0 300 chromium
[82892.878623] [28976] 1003 28976 387993 30432 307 61 0 300 chromium
[82892.878624] [28980] 1003 28980 388564 50813 405 119 0 300 chromium
[82892.878625] [28984] 1003 28984 405454 59656 455 136 0 300 chromium
[82892.878626] [28992] 1003 28992 402263 62772 451 142 0 300 chromium
[82892.878628] [29003] 1003 29003 385571 39699 344 91 0 300 chromium
[82892.878629] [29009] 1003 29009 380822 41385 376 100 0 300 chromium
[82892.878630] [29013] 1003 29013 377475 37963 330 75 0 300 chromium
[82892.878631] [29017] 1003 29017 371606 29948 316 63 0 300 chromium
[82892.878632] [29021] 1003 29021 381843 38634 344 75 0 300 chromium
[82892.878633] [29025] 1003 29025 378780 36394 326 67 0 300 chromium
[82892.878635] [29029] 1003 29029 366039 29309 296 54 0 300 chromium
[82892.878636] [29033] 1003 29033 393296 51117 412 119 0 300 chromium
[82892.878637] [29037] 1003 29037 526665 116851 668 244 0 300 chromium
[82892.878638] [29041] 1003 29041 373636 33414 305 56 0 300 chromium
[82892.878639] [29045] 1003 29045 381594 44951 348 82 0 300 chromium
[82892.878641] [29049] 1003 29049 355096 20913 260 40 0 300 chromium
[82892.878642] [29053] 1003 29053 388600 47046 357 73 0 300 chromium
[82892.878643] [29057] 1003 29057 365874 27205 299 68 0 300 chromium
[82892.878644] [29061] 1003 29061 381530 33668 353 83 0 300 chromium
[82892.878646] [29065] 1003 29065 376219 32128 330 83 0 300 chromium
[82892.878647] [29069] 1003 29069 377903 37948 314 66 0 300 chromium
[82892.878648] [29077] 1003 29077 361333 24819 278 51 0 300 chromium
[82892.878650] [29081] 1003 29081 356061 18384 253 38 0 300 chromium
[82892.878650] [29089] 1003 29089 354038 17488 250 38 0 300 chromium
[82892.878652] [29098] 1003 29098 364877 29854 294 59 0 300 chromium
[82892.878653] [31687] 1003 31687 287404 9711 200 15 0 300 chromium
[82892.878654] [32664] 1003 32664 9158 1542 23 3 0 0 zsh
[82892.878663] [ 7395] 1003 7395 386732 42833 340 62 0 300 chromium
[82892.878664] [31743] 1003 31743 563140 35290 448 7 0 0 mpv
[82892.878668] [10680] 0 10680 17510 1061 38 3 0 0 sudo
[82892.878669] [10717] 0 10717 9752 1615 23 3 0 0 zsh
[82892.878698] [15960] 1003 15960 4331 569 13 3 0 0 screen
[82892.878699] [15962] 1003 15962 4331 568 12 3 0 0 screen
[82892.878700] [15963] 1003 15963 9158 1535 21 3 0 0 zsh
[82892.878728] [18396] 0 18396 17519 1016 39 3 0 0 sudo
[82892.878729] [18397] 1003 18397 1494 166 7 3 0 0 pv
[82892.878730] [18398] 0 18398 17519 1019 37 3 0 0 sudo
[82892.878731] [18399] 0 18399 8815 646 22 3 0 0 zfs
[82892.878732] [18400] 0 18400 8815 637 21 3 0 0 zfs
[82892.878733] [15247] 1003 15247 372659 35514 335 73 0 300 chromium
[82892.878735] Out of memory: Kill process 28960 (chromium) score 319 or sacrifice child
[82892.878758] Killed process 28960 (chromium) total-vm:1987576kB, anon-rss:429952kB, file-rss:53348kB, shmem-rss:152072kB
The worst thing is, I've seen swap usage go up to 3GB at times under heavy load on 4.8.4.1-ck, so my swap definitely works. (vm.swappiness is set to 10, by the way. changing it to the default 60 does not help)
I guess the bug is still alive? Or is there some horrible issue/misconfiguration on my end? I've never had anything like this happen before 4.7.
Offline
Don't focus on the swap, OOM killer can be triggered even if there is free swap, it depends more on the RAM available, in any case I'd guess you have 2 ways to solve this problem :
1) trying to cut down the RAM eating processes in order to save RAM , use the script for that to pin point them. (try understanding if there is a leak or something wrong with them)
2) If you have a free slot just add more RAM and you are done.
I don't think this is a bug as it would be wide spread and oom-killer is triggered only if the RAM is low (talking from experience), even if you see there is free RAM maybe the "next" program tried to eat 500MB and than crashed but was able to trig oom-killer.
However the weird thing is that on both your logs it was Xorg who invoked oom-killer, which is weird as on my list I have different processes using more RAM than Xorg , how much ram it consumes on your installation?
You may find this very informative : https://linux-mm.org/OOM_Killer
Last edited by r0b0t (2016-10-25 12:20:05)
Offline
Don't focus on the swap, OOM killer can be triggered even if there is free swap, it depends more on the RAM available
But this has never happened before 4.7, when some people started having this issue. None of my habits changed when I upgraded to 4.7 or 4.8. But now my Chromium tabs crash sometimes, while in the past they would have simply been swapped off.
Here's the output of the script:
Private + Shared = RAM used Program
80.0 KiB + 12.5 KiB = 92.5 KiB sshtunnel
104.0 KiB + 18.5 KiB = 122.5 KiB chrome-sandbox
208.0 KiB + 65.0 KiB = 273.0 KiB xinit
240.0 KiB + 35.5 KiB = 275.5 KiB xsel
284.0 KiB + 35.0 KiB = 319.0 KiB paxd
300.0 KiB + 94.5 KiB = 394.5 KiB xautolock
360.0 KiB + 100.5 KiB = 460.5 KiB rtkit-daemon
200.0 KiB + 410.5 KiB = 610.5 KiB tmux: client
608.0 KiB + 41.5 KiB = 649.5 KiB crond
396.0 KiB + 308.5 KiB = 704.5 KiB bless
532.0 KiB + 186.5 KiB = 718.5 KiB redshift
412.0 KiB + 308.5 KiB = 720.5 KiB pacserve
424.0 KiB + 305.5 KiB = 729.5 KiB startx
792.0 KiB + 48.5 KiB = 840.5 KiB lvmetad
516.0 KiB + 334.0 KiB = 850.0 KiB systemd-timesyncd
328.0 KiB + 631.0 KiB = 959.0 KiB avahi-daemon (2)
844.0 KiB + 276.5 KiB = 1.1 MiB at-spi-bus-launcher
852.0 KiB + 319.0 KiB = 1.1 MiB gvfsd
804.0 KiB + 389.5 KiB = 1.2 MiB systemd-logind
1.1 MiB + 242.5 KiB = 1.3 MiB sshd
1.0 MiB + 305.0 KiB = 1.3 MiB at-spi2-registryd
1.1 MiB + 281.0 KiB = 1.3 MiB login
1.0 MiB + 384.0 KiB = 1.4 MiB gvfsd-dnssd
1.0 MiB + 401.0 KiB = 1.4 MiB gvfsd-trash
1.0 MiB + 432.5 KiB = 1.5 MiB gvfsd-network
1.1 MiB + 398.0 KiB = 1.5 MiB gvfsd-computer
1.0 MiB + 641.5 KiB = 1.7 MiB (sd-pam)
1.4 MiB + 358.0 KiB = 1.8 MiB sudo
1.7 MiB + 569.5 KiB = 2.2 MiB dbus-daemon (3)
2.2 MiB + 82.5 KiB = 2.3 MiB systemd-udevd
1.6 MiB + 793.0 KiB = 2.3 MiB cups-browsed
1.8 MiB + 590.0 KiB = 2.4 MiB cupsd
2.1 MiB + 422.5 KiB = 2.5 MiB upowerd
1.9 MiB + 608.5 KiB = 2.5 MiB i3bar
2.2 MiB + 396.5 KiB = 2.6 MiB tmux: server
3.2 MiB + 237.5 KiB = 3.4 MiB gvfsd-fuse
2.7 MiB + 707.5 KiB = 3.4 MiB gvfs-udisks2-volume-monitor
3.5 MiB + 116.0 KiB = 3.6 MiB xbindkeys
2.0 MiB + 2.0 MiB = 4.1 MiB systemd (2)
2.7 MiB + 1.4 MiB = 4.1 MiB i3
4.0 MiB + 606.0 KiB = 4.6 MiB colord
4.1 MiB + 439.5 KiB = 4.6 MiB udisksd
4.6 MiB + 1.6 MiB = 6.2 MiB ssh (5)
6.5 MiB + 685.5 KiB = 7.2 MiB NetworkManager
8.0 MiB + 24.0 KiB = 8.1 MiB dhclient
9.6 MiB + 856.0 KiB = 10.4 MiB pulseaudio
11.4 MiB + 393.0 KiB = 11.8 MiB polkitd
9.7 MiB + 2.9 MiB = 12.6 MiB compton
16.9 MiB + 585.5 KiB = 17.5 MiB transmission-daemon
16.2 MiB + 5.3 MiB = 21.5 MiB vim (2)
18.0 MiB + 4.5 MiB = 22.4 MiB pcmanfm
22.8 MiB + 2.1 MiB = 24.8 MiB python3.5 (2)
35.8 MiB + 849.0 KiB = 36.6 MiB urxvt
37.2 MiB + 2.0 MiB = 39.2 MiB zsh (10)
42.6 MiB + 5.8 MiB = 48.4 MiB Xorg
45.0 MiB + 7.9 MiB = 52.9 MiB systemd-journald
79.6 MiB + 2.8 MiB = 82.5 MiB ipython (2)
94.6 MiB + 6.7 MiB = 101.2 MiB Main
157.3 MiB + 6.5 MiB = 163.8 MiB mpv
6.2 GiB + 179.0 MiB = 6.4 GiB chromium (87)
---------------------------------
7.1 GiB
=================================
It's only been 5 hours since boot, so Chromium hasn't gone through its usual course of 9GB yet. I'll post again when uptime gets a little closer to the 80000 seconds mark, at which problems seem to arise.
Offline
I've also been having OOM issues since 4.7, with Xorg as well -- I've had to stay on an LTS kernel where this issue doesn't occur.
Offline
Well chromium using 6.4 GB is quite something, especially on a 8GB max installation. Just out of curiosity how many tabs did you open? Have you tried firefox with the same number of tabs?
In any case if you think this is a bug open it and at least you will get a reply by the devels, however to me it sounds like normal sound behaviour from the kernel.
Offline
Well chromium using 6.4 GB is quite something
No it's not. I have 60-80 tab open (don't judge me) and a few of them are media-heavy like Twitch, which can sometimes eat 1GB apiece. Nothing unusual, lived like that for years. The difference is back then 16GB of RAM + swap was enough, but now my swap usage is at 100KB and stuff is crashing. Somehow it doesn't feel like "normal sound behaviour from the kernel".
Offline
Pages: 1