You are not logged in.

#1 2016-09-27 09:04:11

infinite-recursion
Member
Registered: 2016-09-27
Posts: 4

OOM Issue

Hi,

I am running a home server (Owncloud on LAMP, Samba, Logitech Media Server (AUR)), which has been running without issues for months. A week ago, the server started having out of memory issues. The OOM Killer keeps killing slimserver.pl (Logitech Media Server), mariadb, and httpd. I first noticed this because the media server was down; outwardly, Owncloud seems to run normally, as mariadb and httpd get restarted after they are killed.

I have no idea what is causing this issue. slimserver.pl usually seems to get killed first, but the issues do not stop afterwards, so I think it is a victim rather than the culprit.

The server has 8GB physical memory and a swap partition that hardly seems to get used. This is what free -M gives me (OOM has killed a process a few minutes ago):

                     total        used        free      shared  buff/cache   available
Mem:           7986         425         230           0        7331          7262
Swap:          6143          56        6087

Very little memory is actually listed as 'used', while plenty is available. As I understand, the low 'free' value is due to the fact that the OS uses most of the free space for caching, and thus not relevant here.

At the same time, this is the output of top. Here, a lot of virtual memory seems to be committed (mostly to httpd), but it does not actually seem to get 'used' yet:

top - 10:22:22 up 13:05,  1 user,  load average: 0,07, 0,15, 0,29
Tasks: 233 total,   1 running, 232 sleeping,   0 stopped,   0 zombie
%Cpu0  :   4,0/0,7     5
%Cpu1  :   6,6/0,7     7
GiB Mem :  9,1/7,800  
GiB Swap:  0,9/6,000  

 PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND                                                                                                                                                                                       
    1 root      20   0  195,9m   4,5m   0,0  0,1   0:00.63 S systemd                                                                                                                                                                                     
  204 root      20   0   88,2m  10,0m   0,0  0,1   0:00.99 S  `- systemd-journal                                                                                                                                                                
  238 root      20   0   35,8m   1,9m   0,0  0,0   0:00.10 S  `- systemd-udevd                                                                                                                                                                      
  428 systemd+  20   0  111,5m   2,0m   0,0  0,0   0:00.02 S  `- systemd-timesyn                                                                                                                                                                    
  433 root      20   0   14,2m   2,4m   0,0  0,0   0:00.05 S  `- crond                                                                                                                                                                              
  435 dbus      20   0   32,2m   2,5m   0,0  0,0   0:00.06 S  `- dbus-daemon                                                                                                                                                                        
  447 root      20   0   37,6m   2,7m   0,0  0,0   0:00.04 S  `- systemd-logind                                                                                                                                                                     
  460 root      20   0   36,6m   2,7m   0,0  0,0   0:11.13 S  `- x2gocleansessio                                                                                                                                                                    
  579 root      20   0    6,6m   1,6m   0,0  0,0   0:01.27 S  `- dhcpcd                                                                                                                                                                             
  591 root      20   0  185,5m   3,6m   0,0  0,0   0:00.00 S  `- sddm                                                                                                                                                                               
  623 root      20   0  212,7m   5,5m   0,0  0,1   0:11.83 S      `- Xorg                                                                                                                                                                           
  740 root      20   0  143,3m   4,0m   0,0  0,1   0:00.00 S      `- sddm-helper                                                                                                                                                                    
  748 sddm      20   0  580,2m  13,4m   0,0  0,2   2:40.12 S          `- sddm-greeter                                                                                                                                                               
  602 root      20   0   32,8m   6,0m   0,0  0,1   0:04.53 S  `- ddclient - slee                                                                                                                                                                    
  603 root      20   0   26,6m   0,0m   0,0  0,0   0:00.00 S  `- xrdp-sesman                                                                                                                                                                        
  701 root      20   0  229,5m   3,2m   0,0  0,0   0:02.67 S  `- nmbd                                                                                                                                                                               
  725 root      20   0  311,7m   4,3m   0,0  0,1   0:00.07 S  `- smbd                                                                                                                                                                               
  726 root      20   0  308,0m   1,8m   0,0  0,0   0:00.00 S      `- smbd-notifyd                                                                                                                                                                   
  727 root      20   0  308,0m   2,5m   0,0  0,0   0:00.00 S      `- cleanupd                                                                                                                                                                       
  742 root      20   0  311,7m   2,7m   0,0  0,0   0:00.01 S      `- lpqd                                                                                                                                                                           
  743 sddm      20   0   53,4m   3,6m   0,0  0,0   0:00.03 S  `- systemd                                                                                                                                                                            
  744 sddm      20   0   96,4m   0,0m   0,0  0,0   0:00.00 S      `- (sd-pam)                                                                                                                                                                       
  768 sddm      20   0   32,0m   1,6m   0,0  0,0   0:00.00 S      `- dbus-daemon                                                                                                                                                                    
 4373 root      20   0  327,1m   9,6m   0,0  0,1   0:00.67 S  `- httpd                                                                                                                                                                              
 4374 http      20   0  338,7m  23,8m   0,0  0,3   0:35.46 S      `- httpd                                                                                                                                                                          
 4376 http      20   0  338,7m  24,1m   0,0  0,3   0:39.48 S      `- httpd                                                                                                                                                                          
 7192 http      20   0  338,7m  23,4m   0,0  0,3   0:29.88 S      `- httpd                                                                                                                                                                          
 7213 http      20   0  338,7m  23,8m   0,0  0,3   0:37.98 S      `- httpd                                                                                                                                                                          
15096 http      20   0  338,7m  24,7m   0,0  0,3   0:25.32 S      `- httpd                                                                                                                                                                          
15097 http      20   0  338,7m  25,5m   0,0  0,3   0:24.02 S      `- httpd                                                                                                                                                                          
24381 http      20   0  338,7m  27,4m   0,0  0,3   0:08.96 S      `- httpd                                                                                                                                                                          
25150 http      20   0  338,7m  28,8m   0,0  0,4   0:05.15 S      `- httpd                                                                                                                                                                          
  658 http      20   0  338,6m  27,4m   0,0  0,3   0:02.33 S      `- httpd                                                                                                                                                                          
 8623 http      20   0  334,6m  23,0m   0,0  0,3   0:00.51 S      `- httpd                                                                                                                                                                          
24361 root      20   0   97,8m   6,2m   0,0  0,1   0:00.00 S  `- sshd                                                                                                                                                                               
24379 cube      20   0   97,8m   3,8m   0,0  0,0   0:00.66 S      `- sshd                                                                                                                                                                           
24380 cube      20   0    6,2m   1,4m   0,0  0,0   0:00.00 S          `- sftp-server                                                                                                                                                                
 1362 cube      20   0   17,1m   4,0m   0,0  0,1   0:00.01 S          `- bash                                                                                                                                                                       
10792 cube      20   0   37,9m   4,2m   0,7  0,1   0:00.07 R              `- top                                                                                                                                                                    
24372 cube      20   0   53,4m   4,9m   0,0  0,1   0:00.01 S  `- systemd                                                                                                                                                                            
24374 cube      20   0  176,4m   1,3m   0,0  0,0   0:00.00 S      `- (sd-pam)                                                                                                                                                                       
 6295 mysql     20   0  760,3m 123,5m   0,0  1,5   0:00.73 S  `- mysqld     

And this is the section from the log when the OOM killer first becomes active. The server was rebooted 13 hours ago, ran normally, then 1 hour ago the first process was killed:

Sep 27 09:19:00 agora kernel: x2golistsession invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), order=2, oom_score_adj=0
Sep 27 09:19:00 agora kernel: x2golistsession cpuset=/ mems_allowed=0
Sep 27 09:19:00 agora kernel: CPU: 1 PID: 26540 Comm: x2golistsession Not tainted 4.7.4-1-ARCH #1
Sep 27 09:19:00 agora kernel: Hardware name: System manufacturer System Product Name/P5Q3, BIOS 1001    03/10/2010
Sep 27 09:19:00 agora kernel:  0000000000000286 000000008de34b4a ffff8801bc387b48 ffffffff812ecb02
Sep 27 09:19:00 agora kernel:  ffff8801bc387d28 ffff8800c668dcc0 ffff8801bc387bb8 ffffffff811f6f5c
Sep 27 09:19:00 agora kernel:  ffffffff81178a20 ffff880100000000 ffffffff810fdbe9 ffffffff00000000
Sep 27 09:19:00 agora kernel: Call Trace:
Sep 27 09:19:00 agora kernel:  [<ffffffff812ecb02>] dump_stack+0x63/0x81
Sep 27 09:19:00 agora kernel:  [<ffffffff811f6f5c>] dump_header+0x60/0x1e8
Sep 27 09:19:00 agora kernel:  [<ffffffff81178a20>] ? page_alloc_cpu_notify+0x50/0x50
Sep 27 09:19:00 agora kernel:  [<ffffffff810fdbe9>] ? smp_call_function_single+0xd9/0x140
Sep 27 09:19:00 agora kernel:  [<ffffffff8117635a>] oom_kill_process+0x22a/0x440
Sep 27 09:19:00 agora kernel:  [<ffffffff811769ca>] out_of_memory+0x40a/0x4b0
Sep 27 09:19:00 agora kernel:  [<ffffffff813017e8>] ? find_next_bit+0x18/0x20
Sep 27 09:19:00 agora kernel:  [<ffffffff8117c0bb>] __alloc_pages_nodemask+0xf0b/0xf30
Sep 27 09:19:00 agora kernel:  [<ffffffff8117c434>] alloc_kmem_pages_node+0x54/0xd0
Sep 27 09:19:00 agora kernel:  [<ffffffff81077be6>] copy_process.part.8+0x136/0x19b0
Sep 27 09:19:00 agora kernel:  [<ffffffff811d745c>] ? kmem_cache_alloc+0x1bc/0x1e0
Sep 27 09:19:00 agora kernel:  [<ffffffff811fc69c>] ? get_empty_filp+0x5c/0x1c0
Sep 27 09:19:00 agora kernel:  [<ffffffff811fc716>] ? get_empty_filp+0xd6/0x1c0
Sep 27 09:19:00 agora kernel:  [<ffffffff81079637>] _do_fork+0xd7/0x3d0
Sep 27 09:19:00 agora kernel:  [<ffffffff810799d9>] SyS_clone+0x19/0x20
Sep 27 09:19:00 agora kernel:  [<ffffffff81003c07>] do_syscall_64+0x57/0xb0
Sep 27 09:19:00 agora kernel:  [<ffffffff815d7821>] entry_SYSCALL64_slow_path+0x25/0x25
Sep 27 09:19:00 agora kernel: Mem-Info:
Sep 27 09:19:00 agora kernel: active_anon:53730 inactive_anon:59987 isolated_anon:0
                               active_file:850438 inactive_file:253521 isolated_file:0
                               unevictable:0 dirty:45201 writeback:0 unstable:0
                               slab_reclaimable:730994 slab_unreclaimable:28648
                               mapped:14632 shmem:109 pagetables:3665 bounce:0
                               free:45387 free_pcp:29 free_cma:0
Sep 27 09:19:00 agora kernel: Node 0 DMA free:15896kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:
Sep 27 09:19:00 agora kernel: lowmem_reserve[]: 0 3229 7964 7964
Sep 27 09:19:00 agora kernel: Node 0 DMA32 free:103732kB min:27352kB low:34188kB high:41024kB active_anon:72976kB inactive_anon:93756kB active_file:1340868kB inactive_file:448348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3389
Sep 27 09:19:00 agora kernel: lowmem_reserve[]: 0 0 4734 4734
Sep 27 09:19:00 agora kernel: Node 0 Normal free:61920kB min:40096kB low:50120kB high:60144kB active_anon:141944kB inactive_anon:146192kB active_file:2060884kB inactive_file:565736kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:49
Sep 27 09:19:00 agora kernel: lowmem_reserve[]: 0 0 0 0
Sep 27 09:19:00 agora kernel: Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Sep 27 09:19:00 agora kernel: Node 0 DMA32: 10082*4kB (UME) 7940*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 103848kB
Sep 27 09:19:00 agora kernel: Node 0 Normal: 15243*4kB (UM) 116*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61900kB
Sep 27 09:19:00 agora kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 27 09:19:00 agora kernel: 1106473 total pagecache pages
Sep 27 09:19:00 agora kernel: 2374 pages in swap cache
Sep 27 09:19:00 agora kernel: Swap cache stats: add 49537, delete 47163, find 183880/186643
Sep 27 09:19:00 agora kernel: Free swap  = 6144904kB
Sep 27 09:19:00 agora kernel: Total swap = 6291452kB
Sep 27 09:19:00 agora kernel: 2096653 pages RAM
Sep 27 09:19:00 agora kernel: 0 pages HighMem/MovableOnly
Sep 27 09:19:00 agora kernel: 52027 pages reserved
Sep 27 09:19:00 agora kernel: 0 pages hwpoisoned
Sep 27 09:19:00 agora kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Sep 27 09:19:00 agora kernel: [  204]     0   204    22582     3535      40       3       78             0 systemd-journal
Sep 27 09:19:00 agora kernel: [  238]     0   238     9169      481      21       3      373         -1000 systemd-udevd
Sep 27 09:19:00 agora kernel: [  428]   192   428    28539      510      26       4       99             0 systemd-timesyn
Sep 27 09:19:00 agora kernel: [  433]     0   433     3643      626      13       3      114             0 crond
Sep 27 09:19:00 agora kernel: [  435]    81   435     8251      698      20       3       63          -900 dbus-daemon
Sep 27 09:19:00 agora kernel: [  447]     0   447     9631      814      22       3      111             0 systemd-logind
Sep 27 09:19:00 agora kernel: [  460]     0   460     9365      701      23       3     1008             0 x2gocleansessio
Sep 27 09:19:00 agora kernel: [  579]     0   579     1685      397       9       3       52             0 dhcpcd
Sep 27 09:19:00 agora kernel: [  586]   996   586    87469    57974     176       3      986             0 slimserver.pl
Sep 27 09:19:00 agora kernel: [  591]     0   591    47499      917      47       3      328             0 sddm
Sep 27 09:19:00 agora kernel: [  602]     0   602     8387     1773      19       3      457             0 ddclient - slee
Sep 27 09:19:00 agora kernel: [  603]     0   603     6812        0      16       3      149             0 xrdp-sesman
Sep 27 09:19:00 agora kernel: [  623]     0   623    54440     1416     101       3     1792             0 Xorg
Sep 27 09:19:00 agora kernel: [  688]    89   688   196239    16152     131       4    21369             0 mysqld
Sep 27 09:19:00 agora kernel: [  701]     0   701    58759      838     109       4      422             0 nmbd
Sep 27 09:19:00 agora kernel: [  725]     0   725    79798     1136     151       4      599             0 smbd
Sep 27 09:19:00 agora kernel: [  726]     0   726    78841      458     144       4      613             0 smbd-notifyd
Sep 27 09:19:00 agora kernel: [  727]     0   727    78845      607     143       4      613             0 cleanupd
Sep 27 09:19:00 agora kernel: [  740]     0   740    36681     1040      64       4      328             0 sddm-helper
Sep 27 09:19:00 agora kernel: [  742]     0   742    79794      705     146       4      593             0 lpqd
Sep 27 09:19:00 agora kernel: [  743]   997   743    13670     1095      30       3      150             0 systemd
Sep 27 09:19:00 agora kernel: [  744]   997   744    24688        9      49       3      419             0 (sd-pam)
Sep 27 09:19:00 agora kernel: [  748]   997   748   148414     3297     169       3     4943             0 sddm-greeter
Sep 27 09:19:00 agora kernel: [  768]   997   768     8186      417      19       3       80             0 dbus-daemon
Sep 27 09:19:00 agora kernel: [ 4373]     0  4373    83725     2486     138       3     1513             0 httpd
Sep 27 09:19:00 agora kernel: [ 4374]    33  4374    86718     6065     140       3      818             0 httpd
Sep 27 09:19:00 agora kernel: [ 4375]    33  4375    86716     5939     140       3      812             0 httpd
Sep 27 09:19:00 agora kernel: [ 4376]    33  4376    86716     6155     140       3      807             0 httpd
Sep 27 09:19:00 agora kernel: [ 7192]    33  7192    86718     5986     140       3      819             0 httpd
Sep 27 09:19:00 agora kernel: [ 7213]    33  7213    86717     5939     140       3      812             0 httpd
Sep 27 09:19:00 agora kernel: [15096]    33 15096    86717     6409     140       3      774             0 httpd
Sep 27 09:19:00 agora kernel: [15097]    33 15097    86670     5984     139       3      779             0 httpd
Sep 27 09:19:00 agora kernel: [24361]     0 24361    25045     1703      52       3        0             0 sshd
Sep 27 09:19:00 agora kernel: [24372]  1000 24372    13670     1504      30       3        0             0 systemd
Sep 27 09:19:00 agora kernel: [24374]  1000 24374    45170      333      52       3      111             0 (sd-pam)
Sep 27 09:19:00 agora kernel: [24379]  1000 24379    25045      851      49       3        0             0 sshd
Sep 27 09:19:00 agora kernel: [24380]  1000 24380     1590      382       9       3        0             0 sftp-server
Sep 27 09:19:00 agora kernel: [24381]    33 24381    86670     6704     139       3      720             0 httpd
Sep 27 09:19:00 agora kernel: [25150]    33 25150    85134     5358     136       3      737             0 httpd
Sep 27 09:19:00 agora kernel: [25210]    33 25210    85134     5353     136       3      737             0 httpd
Sep 27 09:19:00 agora kernel: [25560]  1001 25560    82889     5024     157       4      329             0 smbd
Sep 27 09:19:00 agora kernel: [26539]     0 26539     4319      860      13       3        0             0 sh
Sep 27 09:19:00 agora kernel: [26540]     0 26540    13129     3447      31       3        0             0 x2golistsession
Sep 27 09:19:00 agora kernel: Out of memory: Kill process 586 (slimserver.pl) score 16 or sacrifice child
Sep 27 09:19:00 agora kernel: Killed process 586 (slimserver.pl) total-vm:349876kB, anon-rss:224640kB, file-rss:7256kB, shmem-rss:0kB
Sep 27 09:19:00 agora systemd[1]: logitechmediaserver.service: Main process exited, code=killed, status=9/KILL
Sep 27 09:19:00 agora systemd[1]: logitechmediaserver.service: Unit entered failed state.
Sep 27 09:19:00 agora systemd[1]: logitechmediaserver.service: Failed with result 'signal'.

Can anybody give me a pointer on where to start looking based on this? I do not even understand which type of memory is depleted - the process list does not look fundamentally different from the output of top, i.e. I do not see a process doing anything unusual.

Thank you!

Offline

#2 2016-09-27 10:19:07

fturco
Member
Registered: 2010-07-12
Posts: 40

Re: OOM Issue

There is a bug with kernel 4.7.x and OOM killer. As far as I know it has been fixed with version 4.7.5. Which kernel version do you have?

Offline

#3 2016-09-27 13:02:28

infinite-recursion
Member
Registered: 2016-09-27
Posts: 4

Re: OOM Issue

Thanks fturco, that is probably it. I'm on the core repository, so currently kernel 4.7.4-1.

This post then seems to discuss the same issue: https://bbs.archlinux.org/viewtopic.php?id=216716

A patch concerning this was discussed on LKML and seems to have been included in version 4.7.5-1, so I suppose I will just wait for this to move into core.

Offline

#4 2016-09-27 13:22:15

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: OOM Issue

You can use this script to see who is using all the RAM (it lists nicely beginning with the less RAM eater to the most calculating also private + shared  :

#!/usr/bin/env python
import getopt
import time
import errno
import os
import sys

try:
    # md5 module is deprecated on python 2.6
    # so try the newer hashlib first
    import hashlib
    md5_new = hashlib.md5
except ImportError:
    import md5
    md5_new = md5.new


# The following exits cleanly on Ctrl-C or EPIPE
# while treating other exceptions as before.
def std_exceptions(etype, value, tb):
    sys.excepthook = sys.__excepthook__
    if issubclass(etype, KeyboardInterrupt):
        pass
    elif issubclass(etype, IOError) and value.errno == errno.EPIPE:
        pass
    else:
        sys.__excepthook__(etype, value, tb)
sys.excepthook = std_exceptions

#
#   Define some global variables
#
uname = os.uname()
if uname[0] == "FreeBSD":
    proc = "/compat/linux/proc/"
else:
    proc = "/proc/"

PAGESIZE = os.sysconf("SC_PAGE_SIZE") / 1024 #KiB
our_pid = os.getpid()

have_pss = 0

#
#   Functions
#

def parse_options():
    try:
        long_options = ['split-args', 'help']
        opts, args = getopt.getopt(sys.argv[1:], "shp:w:", long_options)
    except getopt.GetoptError, e:
        print help()
        sys.exit(3)

    # ps_mem.py options
    split_args = False
    pids_to_show = None
    watch = None

    for o, a in opts:
        if o in ('-s', '--split-args'):
            split_args = True
        if o in ('-h', '--help'):
            print help()
            sys.exit(0)
        if o in ('-p',):
            try:
                pids_to_show = [int(x) for x in a.split(',')]
            except:
                print help()
                sys.exit(3)
        if o in ('-w',):
            try:
                watch = int(a)
            except:
                print help()
                sys.exit(3)

    return (split_args, pids_to_show, watch)

def help():
    help_msg = 'ps_mem.py - Show process memory usage\n'\
    '\n'\
    '-h                                 Show this help\n'\
    '-w <N>                             Measure and show process memory every N seconds\n'\
    '-p <pid>[,pid2,...pidN]            Only show memory usage PIDs in the specified list\n'

    return help_msg

#(major,minor,release)
def kernel_ver():
    kv = open(proc + "sys/kernel/osrelease", "rt").readline().split(".")[:3]
    last = len(kv)
    if last == 2:
        kv.append('0')
    last -= 1
    for char in "-_":
        kv[last] = kv[last].split(char)[0]
    try:
        int(kv[last])
    except:
        kv[last] = 0
    return (int(kv[0]), int(kv[1]), int(kv[2]))


#return Private,Shared
#Note shared is always a subset of rss (trs is not always)
def getMemStats(pid):
    global have_pss
    mem_id = pid #unique
    Private_lines = []
    Shared_lines = []
    Pss_lines = []
    Rss = (int(open(proc + str(pid) + "/statm", "rt").readline().split()[1])
           * PAGESIZE)
    if os.path.exists(proc + str(pid) + "/smaps"): #stat
        digester = md5_new()
        for line in open(proc + str(pid) + "/smaps", "rb").readlines(): #open
            # Note we checksum smaps as maps is usually but
            # not always different for separate processes.
            digester.update(line)
            line = line.decode("ascii")
            if line.startswith("Shared"):
                Shared_lines.append(line)
            elif line.startswith("Private"):
                Private_lines.append(line)
            elif line.startswith("Pss"):
                have_pss = 1
                Pss_lines.append(line)
        mem_id = digester.hexdigest()
        Shared = sum([int(line.split()[1]) for line in Shared_lines])
        Private = sum([int(line.split()[1]) for line in Private_lines])
        #Note Shared + Private = Rss above
        #The Rss in smaps includes video card mem etc.
        if have_pss:
            pss_adjust = 0.5 # add 0.5KiB as this avg error due to trunctation
            Pss = sum([float(line.split()[1])+pss_adjust for line in Pss_lines])
            Shared = Pss - Private
    elif (2,6,1) <= kernel_ver() <= (2,6,9):
        Shared = 0 #lots of overestimation, but what can we do?
        Private = Rss
    else:
        Shared = int(open(proc+str(pid)+"/statm", "rt").readline().split()[2])
        Shared *= PAGESIZE
        Private = Rss - Shared
    return (Private, Shared, mem_id)


def getCmdName(pid, split_args):
    cmdline = open(proc+"%d/cmdline" % pid, "rt").read().split("\0")
    if cmdline[-1] == '' and len(cmdline) > 1:
        cmdline = cmdline[:-1]
    path = os.path.realpath(proc+"%d/exe" % pid) #exception for kernel threads
    if split_args:
        return " ".join(cmdline)
    if path.endswith(" (deleted)"):
        path = path[:-10]
        if os.path.exists(path):
            path += " [updated]"
        else:
            #The path could be have prelink stuff so try cmdline
            #which might have the full path present. This helped for:
            #/usr/libexec/notification-area-applet.#prelink#.fX7LCT (deleted)
            if os.path.exists(cmdline[0]):
                path = cmdline[0] + " [updated]"
            else:
                path += " [deleted]"
    exe = os.path.basename(path)
    cmd = open(proc+"%d/status" % pid, "rt").readline()[6:-1]
    if exe.startswith(cmd):
        cmd = exe #show non truncated version
        #Note because we show the non truncated name
        #one can have separated programs as follows:
        #584.0 KiB +   1.0 MiB =   1.6 MiB    mozilla-thunder (exe -> bash)
        # 56.0 MiB +  22.2 MiB =  78.2 MiB    mozilla-thunderbird-bin
    return cmd


#The following matches "du -h" output
#see also human.py
def human(num, power="Ki"):
    powers = ["Ki", "Mi", "Gi", "Ti"]
    while num >= 1000: #4 digits
        num /= 1024.0
        power = powers[powers.index(power)+1]
    return "%.1f %s" % (num, power)


def cmd_with_count(cmd, count):
    if count > 1:
        return "%s (%u)" % (cmd, count)
    else:
        return cmd

#Warn of possible inaccuracies
#2 = accurate & can total
#1 = accurate only considering each process in isolation
#0 = some shared mem not reported
#-1= all shared mem not reported
def shared_val_accuracy():
    """http://wiki.apache.org/spamassassin/TopSharedMemoryBug"""
    kv = kernel_ver()
    if kv[:2] == (2,4):
        if open(proc+"meminfo", "rt").read().find("Inact_") == -1:
            return 1
        return 0
    elif kv[:2] == (2,6):
        pid = str(os.getpid())
        if os.path.exists(proc+pid+"/smaps"):
            if open(proc+pid+"/smaps", "rt").read().find("Pss:")!=-1:
                return 2
            else:
                return 1
        if (2,6,1) <= kv <= (2,6,9):
            return -1
        return 0
    elif kv[0] > 2:
        return 2
    else:
        return 1

def show_shared_val_accuracy( possible_inacc ):
    if possible_inacc == -1:
        sys.stderr.write(
         "Warning: Shared memory is not reported by this system.\n"
        )
        sys.stderr.write(
         "Values reported will be too large, and totals are not reported\n"
        )
    elif possible_inacc == 0:
        sys.stderr.write(
         "Warning: Shared memory is not reported accurately by this system.\n"
        )
        sys.stderr.write(
         "Values reported could be too large, and totals are not reported\n"
        )
    elif possible_inacc == 1:
        sys.stderr.write(
         "Warning: Shared memory is slightly over-estimated by this system\n"
         "for each program, so totals are not reported.\n"
        )
    sys.stderr.close()

def get_memory_usage( pids_to_show, split_args, include_self=False, only_self=False ):
    cmds = {}
    shareds = {}
    mem_ids = {}
    count = {}
    for pid in os.listdir(proc):
        if not pid.isdigit():
            continue
        pid = int(pid)
        
        # Some filters
        if only_self and pid != our_pid:
            continue
        if pid == our_pid and not include_self:
            continue
        if pids_to_show is not None and pid not in pids_to_show:
            continue
        
        try:
            cmd = getCmdName(pid, split_args)
        except:
            #permission denied or
            #kernel threads don't have exe links or
            #process gone
            continue
        
        try:
            private, shared, mem_id = getMemStats(pid)
        except:
            continue #process gone
        if shareds.get(cmd):
            if have_pss: #add shared portion of PSS together
                shareds[cmd] += shared
            elif shareds[cmd] < shared: #just take largest shared val
                shareds[cmd] = shared
        else:
            shareds[cmd] = shared
        cmds[cmd] = cmds.setdefault(cmd, 0) + private
        if cmd in count:
            count[cmd] += 1
        else:
            count[cmd] = 1
        mem_ids.setdefault(cmd, {}).update({mem_id:None})

    #Add shared mem for each program
    total = 0
    for cmd in cmds:
        cmd_count = count[cmd]
        if len(mem_ids[cmd]) == 1 and cmd_count > 1:
            # Assume this program is using CLONE_VM without CLONE_THREAD
            # so only account for one of the processes
            cmds[cmd] /= cmd_count
            if have_pss:
                shareds[cmd] /= cmd_count
        cmds[cmd] = cmds[cmd] + shareds[cmd]
        total += cmds[cmd] #valid if PSS available

    sorted_cmds = cmds.items()
    sorted_cmds.sort(lambda x, y:cmp(x[1], y[1]))
    sorted_cmds = [x for x in sorted_cmds if x[1]]

    return sorted_cmds, shareds, count, total

def print_header():
    sys.stdout.write(" Private  +   Shared  =  RAM used\tProgram \n\n")

def print_memory_usage(sorted_cmds, shareds, count, total):
    for cmd in sorted_cmds:
        sys.stdout.write("%8sB + %8sB = %8sB\t%s\n" %
                         (human(cmd[1]-shareds[cmd[0]]),
                          human(shareds[cmd[0]]), human(cmd[1]),
                          cmd_with_count(cmd[0], count[cmd[0]])))
    if have_pss:
        sys.stdout.write("%s\n%s%8sB\n%s\n" %
                         ("-" * 33, " " * 24, human(total), "=" * 33))

def verify_environment():
    if os.geteuid() != 0:
        sys.stderr.write("Sorry, root permission required.\n")
        if __name__ == '__main__':
            sys.stderr.close()
            sys.exit(1)

    try:
        kv = kernel_ver()
    except (IOError, OSError):
        val = sys.exc_info()[1]
        if val.errno == errno.ENOENT:
            sys.stderr.write(
              "Couldn't access /proc\n"
              "Only GNU/Linux and FreeBSD (with linprocfs) are supported\n")
            sys.exit(2)
        else:
            raise

if __name__ == '__main__':
    verify_environment()
    split_args, pids_to_show, watch = parse_options()

    print_header()

    if watch is not None:
        try:
            sorted_cmds = True
            while sorted_cmds:
                sorted_cmds, shareds, count, total = get_memory_usage( pids_to_show, split_args )
                print_memory_usage(sorted_cmds, shareds, count, total)
                time.sleep(watch)
            else:
                sys.stdout.write('Process does not exist anymore.\n')
        except KeyboardInterrupt:
            pass
    else:
        # This is the default behavior
        sorted_cmds, shareds, count, total = get_memory_usage( pids_to_show, split_args )
        print_memory_usage(sorted_cmds, shareds, count, total)


    # We must close explicitly, so that any EPIPE exception
    # is handled by our excepthook, rather than the default
    # one which is reenabled after this script finishes.
    sys.stdout.close()

    vm_accuracy = shared_val_accuracy()
    show_shared_val_accuracy( vm_accuracy )

Try to run it when you see the first instance of oom-killer, if really there is no RAM left than it's probably the bug. I'd double check though.
Remember to lsof -p on the RAM eater if you find it.
It may happen during peak hours.

Offline

#5 2016-09-29 00:05:16

infinite-recursion
Member
Registered: 2016-09-27
Posts: 4

Re: OOM Issue

The issue seems solved, but remains mysterious: After doing another full system upgrade this morning and rebooting, the issue did not reappear today although I was still on kernel 4.7.4-1 (4.7.5-1 was already listed as core here on the website this morning, but apparently there is a delay between what is listed here and the mirror I am using - the update has only reached me just now).

r0b0t, thanks for the script. It crashes when I try to run it, though - which version of Python does this require?

Offline

#6 2016-09-29 00:12:00

atomicbeef
Member
Registered: 2015-09-30
Posts: 98

Re: OOM Issue

r0b0t's script is written in Python 2, so try running it with a Python 2 interpreter and see if it crashes.

Offline

#7 2016-09-29 00:22:28

loqs
Member
Registered: 2014-03-06
Posts: 17,197

Re: OOM Issue

From r0b0t's script

#!/usr/bin/env python

Yet

$ /usr/bin/env python
Python 3.5.2 (default, Jun 28 2016, 08:46:01) 
[GCC 6.1.1 20160602] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
pacman -Qo /usr/bin/python
/usr/bin/python is owned by python 3.5.2-1

but

        print help()

Is definately python2 not python3
Edit:
4.7.5-1 is now in core hopefully resolving the issue for you.

Last edited by loqs (2016-09-29 01:10:38)

Offline

#8 2016-09-30 12:36:58

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: OOM Issue

Hi,

sorry forgot to point it out, it's python2 , you can check the top RAM eaters also with :

ps auxf | sort -nr -k 4 | head -100
ps -eo rss,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | egrep -v 0.00 | sort -n

You can find the original script here : https://github.com/pixelb/ps_mem/blob/master/ps_mem.py (props to pixelb)

Offline

#9 2016-10-24 15:55:15

bus
Member
Registered: 2014-10-12
Posts: 25

Re: OOM Issue

I thought these issues have been fixed, but I keep getting random OOM kills while only 100KB of 16GB swap is used.
This happens both with completely stock kernel and linux-ck (from the repo-ck).

Here's the dmesg log from three days ago of two consecutive kills on the stock linux 4.8.2.1:

[88664.647172] Xorg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
[88664.647176] Xorg cpuset=/ mems_allowed=0
[88664.647180] CPU: 0 PID: 953 Comm: Xorg Tainted: P           O    4.8.2-1-ARCH #1
[88664.647181] Hardware name: System manufacturer System Product Name/P8P67 REV 3.1, BIOS 3602 11/01/2012
[88664.647182]  0000000000000286 0000000011eed91e ffff8803f875b978 ffffffff812fe280
[88664.647184]  ffff8803f875bb38 ffff880379bb3900 ffff8803f875b9e8 ffffffff81205f32
[88664.647185]  ffff8803f875b980 0000000000000000 0000000000000000 0000000000000000
[88664.647186] Call Trace:
[88664.647191]  [<ffffffff812fe280>] dump_stack+0x63/0x83
[88664.647194]  [<ffffffff81205f32>] dump_header+0x5c/0x1ea
[88664.647195]  [<ffffffff8117ca25>] oom_kill_process+0x265/0x410
[88664.647197]  [<ffffffff810864c7>] ? has_capability_noaudit+0x17/0x20
[88664.647198]  [<ffffffff8117cfa0>] out_of_memory+0x380/0x420
[88664.647201]  [<ffffffff81313118>] ? find_next_bit+0x18/0x20
[88664.647202]  [<ffffffff81182600>] __alloc_pages_nodemask+0xda0/0xde0
[88664.647205]  [<ffffffff811d6585>] alloc_pages_current+0x95/0x140
[88664.647206]  [<ffffffff811a3e3e>] kmalloc_order_trace+0x2e/0xf0
[88664.647207]  [<ffffffff811e1eca>] __kmalloc+0x1ea/0x200
[88664.647218]  [<ffffffffa17a02c7>] nvkms_alloc+0x27/0x60 [nvidia_modeset]
[88664.647222]  [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88664.647228]  [<ffffffffa17daf2a>] _nv001929kms+0x1a/0x30 [nvidia_modeset]
[88664.647234]  [<ffffffffa17c7282>] ? _nv001878kms+0x32/0xcf0 [nvidia_modeset]
[88664.647235]  [<ffffffff811e1eca>] ? __kmalloc+0x1ea/0x200
[88664.647236]  [<ffffffff81205d54>] ? __check_object_size+0x54/0x1d6
[88664.647240]  [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88664.647244]  [<ffffffffa17a15ae>] ? _nv000169kms+0x2e/0x40 [nvidia_modeset]
[88664.647248]  [<ffffffffa17a2101>] ? nvKmsIoctl+0x161/0x1e0 [nvidia_modeset]
[88664.647253]  [<ffffffffa17a0da5>] ? nvkms_ioctl_common+0x45/0x80 [nvidia_modeset]
[88664.647257]  [<ffffffffa17a0e51>] ? nvkms_ioctl+0x71/0xa0 [nvidia_modeset]
[88664.647391]  [<ffffffffa0b7a080>] ? nvidia_frontend_compat_ioctl+0x40/0x50 [nvidia]
[88664.647431]  [<ffffffffa0b7a09e>] ? nvidia_frontend_unlocked_ioctl+0xe/0x10 [nvidia]
[88664.647433]  [<ffffffff8121c313>] ? do_vfs_ioctl+0xa3/0x5f0
[88664.647434]  [<ffffffff81088f9b>] ? recalc_sigpending+0x1b/0x50
[88664.647435]  [<ffffffff81089af1>] ? __set_task_blocked+0x41/0xa0
[88664.647436]  [<ffffffff8121c8d9>] ? SyS_ioctl+0x79/0x90
[88664.647437]  [<ffffffff8108c644>] ? SyS_rt_sigprocmask+0x74/0xc0
[88664.647439]  [<ffffffff815f7cf2>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
[88664.647440] Mem-Info:
[88664.647443] active_anon:2772859 inactive_anon:277286 isolated_anon:0
                active_file:453729 inactive_file:113612 isolated_file:0
                unevictable:0 dirty:40975 writeback:0 unstable:0
                slab_reclaimable:313689 slab_unreclaimable:24852
                mapped:135832 shmem:70854 pagetables:33894 bounce:0
                free:33374 free_pcp:144 free_cma:0
[88664.647445] Node 0 active_anon:11091436kB inactive_anon:1109144kB active_file:1814916kB inactive_file:454448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:543328kB dirty:163900kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 4982784kB anon_thp: 283416kB writeback_tmp:0kB unstable:0kB pages_scanned:55 all_unreclaimable? no
[88664.647446] Node 0 DMA free:15888kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88664.647448] lowmem_reserve[]: 0 3463 15989 15989 15989
[88664.647449] Node 0 DMA32 free:64616kB min:14620kB low:18272kB high:21924kB active_anon:2282848kB inactive_anon:56672kB active_file:579772kB inactive_file:170876kB unevictable:0kB writepending:18308kB present:3636768kB managed:3561020kB mlocked:0kB slab_reclaimable:333748kB slab_unreclaimable:18808kB kernel_stack:96kB pagetables:4372kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88664.647451] lowmem_reserve[]: 0 0 12526 12526 12526
[88664.647453] Node 0 Normal free:52992kB min:52892kB low:66112kB high:79332kB active_anon:8808588kB inactive_anon:1052472kB active_file:1235080kB inactive_file:283652kB unevictable:0kB writepending:145620kB present:13090816kB managed:12827296kB mlocked:0kB slab_reclaimable:921008kB slab_unreclaimable:80588kB kernel_stack:21472kB pagetables:131204kB bounce:0kB free_pcp:540kB local_pcp:0kB free_cma:0kB
[88664.647455] lowmem_reserve[]: 0 0 0 0 0
[88664.647456] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB
[88664.647462] Node 0 DMA32: 11181*4kB (UME) 2322*8kB (UME) 93*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64788kB
[88664.647466] Node 0 Normal: 3675*4kB (MEH) 4740*8kB (UMH) 0*16kB 3*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52844kB
[88664.647471] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[88664.647471] 638252 total pagecache pages
[88664.647472] 4 pages in swap cache
[88664.647473] Swap cache stats: add 94, delete 90, find 0/0
[88664.647473] Free swap  = 16776844kB
[88664.647474] Total swap = 16777212kB
[88664.647474] 4185892 pages RAM
[88664.647474] 0 pages HighMem/MovableOnly
[88664.647474] 84838 pages reserved
[88664.647475] 0 pages hwpoisoned
[88664.647475] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[88664.647482] [  357]     0   357    38691     7785      68       3        0             0 systemd-journal
[88664.647486] [  465]     0   465     3032      539      11       3        0             0 paxd
[88664.647487] [  474]     0   474     9141      947      19       3        0         -1000 systemd-udevd
[88664.647489] [  647]     0   647    43505      360      18       3        0             0 lvmetad
[88664.647491] [  780]   192   780    28539      970      26       3        0             0 systemd-timesyn
[88664.647492] [  784]    84   784    10327      835      24       3        0             0 avahi-daemon
[88664.647493] [  785]    84   785    10295       77      23       3        0             0 avahi-daemon
[88664.647494] [  792]     0   792     3259      642      12       3        0             0 crond
[88664.647495] [  793]     0   793     9629     1123      25       3        0             0 systemd-logind
[88664.647496] [  796]    81   796     8355      995      21       3        0          -900 dbus-daemon
[88664.647497] [  814]     0   814   108612     3521      74       4        0             0 NetworkManager
[88664.647498] [  821]  1003   821   178158     6659      65       4        0             0 transmission-da
[88664.647499] [  826]     0   826    21924     1682      47       3        0             0 cupsd
[88664.647500] [  829]     0   829    10100      953      25       3        0         -1000 sshd
[88664.647500] [  831]     0   831    22532      945      49       3        0             0 login
[88664.647501] [  833]   124   833    76298     2522      51       4        0             0 colord
[88664.647502] [  838]     0   838    64689     2320      60       4        0             0 cups-browsed
[88664.647503] [  844]   102   844   131172     3821      53       4        0             0 polkitd
[88664.647504] [  880]     0   880     3702     2182      12       3        0             0 dhclient
[88664.647505] [  909]  1003   909    13724     1551      30       3        0             0 systemd
[88664.647506] [  910]  1003   910    24712      461      49       3        0             0 (sd-pam)
[88664.647507] [  917]  1003   917     3406      781      11       3        0             0 startx
[88664.647508] [  952]  1003   952     3979      228      15       3        0             0 xinit
[88664.647510] [  953]  1003   953    61019    14216      93       4        0             0 Xorg
[88664.647511] [  963]  1003   963    32606     3724      66       4        0             0 i3
[88664.647513] [  967]  1003   967     8219      738      21       3        0             0 dbus-daemon
[88664.647515] [  973]  1003   973    15053     1103      32       3        0             0 redshift
[88664.647516] [  979]  1003   979     5029      252      15       3        0             0 xautolock
[88664.647518] [  985]  1003   985    28419     4532      59       4        0             0 compton
[88664.647519] [  987]  1003   987    27077     1967      56       3        0             0 i3bar
[88664.647520] [  990]  1003   990   292303    11908      65       5        0             0 urxvt
[88664.647522] [  992]  1003   992   207220    22365     163       4        0             0 pcmanfm
[88664.647523] [  993]  1003   993   189775     3626      45       4        0             0 python3
[88664.647524] [  997]  1003   997   139110     3981     111       3        0             0 pulseaudio
[88664.647525] [ 1007]  1003  1007    18073     1449      29       3        0             0 xbindkeys
[88664.647526] [ 1008]   133  1008    44462      657      22       4        0             0 rtkit-daemon
[88664.647528] [ 1017]  1003  1017    69992     1508      36       4        0             0 gvfsd
[88664.647529] [ 1022]  1003  1022   103333     1345      36       4        0             0 gvfsd-fuse
[88664.647530] [ 1031]  1003  1031     4649      819      13       3        0             0 zsh
[88664.647531] [ 1035]  1003  1035     4887      714      14       3        0             0 tmux: client
[88664.647532] [ 1039]  1003  1039     6481     1830      17       3        0             0 tmux: server
[88664.647533] [ 1041]  1003  1041     3406      721      11       3        0             0 sh
[88664.647535] [ 1043]  1003  1043     9061     1699      23       3        0             0 zsh
[88664.647536] [ 1171]  1003  1171   166203     2876      62       4        0             0 gvfs-udisks2-vo
[88664.647537] [ 1217]     0  1217   116008     2325      52       4        0             0 udisksd
[88664.647538] [ 1371]  1003  1371    88997     1529      40       3        0             0 gvfsd-trash
[88664.647539] [ 1380]  1003  1380    89537     2046      41       4        0             0 gvfsd-computer
[88664.647540] [ 1387]  1003  1387   107970     1572      45       4        0             0 gvfsd-network
[88664.647541] [ 1393]  1003  1393    93102     1635      49       3        0             0 gvfsd-dnssd
[88664.647542] [ 1455]  1003  1455    49157     3350      94       3        0             0 vim
[88664.647543] [ 1710]  1003  1710     9779     1974      24       3        0             0 zsh
[88664.647544] [ 1742]  1003  1742     1039       17       8       3        0             0 sshtunnel
[88664.647545] [ 1744]  1003  1744     9646     1584      24       3        0             0 ssh
[88664.647546] [ 1807]  1003  1807     9779     1918      23       3        0             0 zsh
[88664.647547] [ 1904]  1003  1904     9518     1331      23       3        0             0 ssh
[88664.647548] [ 1932]  1003  1932     9568      707      22       3        0             0 ssh
[88664.647549] [ 1983]    99  1983     3406      684      13       3        0             0 pacserve
[88664.647550] [ 1984]    99  1984    90540     5776      78       4        0             0 python3
[88664.647550] [ 2013]  1003  2013     9518     1339      22       3        0             0 ssh
[88664.647551] [ 2017]  1003  2017     9546      625      21       3        0             0 ssh
[88664.647552] [ 2263]  1003  2263    10288     2328      24       3        0             0 zsh
[88664.647553] [ 2609]  1003  2609   475399   128767     602       6        0             0 chromium
[88664.647554] [ 2611]  1003  2611     1585      193       8       3        0             0 chrome-sandbox
[88664.647555] [ 2612]  1003  2612   115217    11933     170       4        0             0 chromium
[88664.647556] [ 2618]  1003  2618     1585      183       8       3        0             0 chrome-sandbox
[88664.647557] [ 2619]  1003  2619     7913     1108      22       4        0             0 nacl_helper
[88664.647558] [ 2623]  1003  2623   115217     2880     130       4        0             0 chromium
[88664.647559] [ 2706]  1003  2706   282557   128793     517       5        0           200 chromium
[88664.647560] [ 2712]  1003  2712   116433     6091     148       4        0           200 chromium
[88664.647561] [ 2720]  1003  2720   194399     9559     174      11        0           300 chromium
[88664.647562] [ 2726]  1003  2726   215159    17706     224      29        0           300 chromium
[88664.647563] [ 2742]  1003  2742   279001    63997     377      63        0           300 chromium
[88664.647564] [ 2750]  1003  2750   300872    76896     455     144        0           300 chromium
[88664.647565] [ 2754]  1003  2754   222911    24174     253      44        0           300 chromium
[88664.647566] [ 2762]  1003  2762   307503    99537     444      85        0           300 chromium
[88664.647567] [ 2771]  1003  2771   219930    20678     235      34        0           300 chromium
[88664.647567] [ 2835]  1003  2835   215415    18520     225      29        0           300 chromium
[88664.647569] [ 2839]  1003  2839   215415    17775     224      29        0           300 chromium
[88664.647570] [ 2866]  1003  2866   222592    21000     233      32        0           300 chromium
[88664.647571] [ 2886]  1003  2886   265074    43236     317      67        0           300 chromium
[88664.647571] [ 2896]  1003  2896   235280    31906     265      40        0           300 chromium
[88664.647572] [ 2899]  1003  2899   221167    22566     243      36        0           300 chromium
[88664.647573] [ 2905]  1003  2905   260572    49268     313      64        0           300 chromium
[88664.647574] [ 2911]  1003  2911   227028    28466     250      34        0           300 chromium
[88664.647575] [ 2928]  1003  2928   243407    33841     305      61        0           300 chromium
[88664.647577] [ 2932]  1003  2932   218563    20383     228      30        0           300 chromium
[88664.647578] [ 2943]  1003  2943   218816    21178     236      31        0           300 chromium
[88664.647579] [ 2947]  1003  2947   216752    20447     228      29        0           300 chromium
[88664.647580] [ 2954]  1003  2954   237878    29705     269      44        0           300 chromium
[88664.647581] [ 2958]  1003  2958   286159    77251     464      93        0           300 chromium
[88664.647582] [ 2987]  1003  2987   207988    17274     218      28        0           300 chromium
[88664.647583] [ 2993]  1003  2993   212086    17521     221      29        0           300 chromium
[88664.647583] [ 3004]  1003  3004   214646    19376     228      32        0           300 chromium
[88664.647584] [ 3008]  1003  3008   207988    18075     219      29        0           300 chromium
[88664.647585] [ 3021]  1003  3021   236335    30025     267      52        0           300 chromium
[88664.647587] [ 3042]  1003  3042   240257    34147     277      59        0           300 chromium
[88664.647588] [ 3133]     0  3133    75065     2029      45       3        0             0 upowerd
[88664.647589] [ 3250]  1003  3250   274553    68585     382      70        0           300 chromium
[88664.647591] [ 3292]  1003  3292   507638   185703     998     294        0           300 chromium
[88664.647592] [ 3328]  1003  3328   388671   164123     832     356        0           300 chromium
[88664.647593] [ 3415]  1003  3415   307696    88210     456     127        0           300 chromium
[88664.647595] [ 3536]  1003  3536   282332    78604     409     107        0           300 chromium
[88664.647596] [ 3546]  1003  3546   305499   101499     511     175        0           300 chromium
[88664.647598] [ 3674]  1003  3674   224280    25328     255      45        0           300 chromium
[88664.647599] [ 3694]  1003  3694   245342    43043     317      74        0           300 chromium
[88664.647600] [ 3747]  1003  3747   219705    24701     253      43        0           300 chromium
[88664.647601] [ 3756]  1003  3756   483482   240890    1059     340        0           300 chromium
[88664.647603] [ 3806]  1003  3806   360885   131138     629     210        0           300 chromium
[88664.647603] [ 3866]  1003  3866   283736    71387     428     120        0           300 chromium
[88664.647604] [ 3875]  1003  3875   424606   200196     769     242        0           300 chromium
[88664.647606] [ 3981]  1003  3981   372987   131164     673     223        0           300 chromium
[88664.647607] [ 4227]  1003  4227   230134    29402     269      51        0           300 chromium
[88664.647608] [ 4291]  1003  4291   288571    78957     435      90        0           300 chromium
[88664.647609] [ 4472]  1003  4472   282333    81069     413     107        0           300 chromium
[88664.647610] [ 4576]  1003  4576   277599    77367     405     116        0           300 chromium
[88664.647611] [ 4694]  1003  4694   232276    31002     274      49        0           300 chromium
[88664.647613] [ 4702]  1003  4702   256330    44934     335      82        0           300 chromium
[88664.647613] [ 4755]  1003  4755   262128    51582     351      83        0           300 chromium
[88664.647615] [ 4773]  1003  4773   222826    25658     254      43        0           300 chromium
[88664.647615] [ 4793]  1003  4793   245184    39982     310      75        0           300 chromium
[88664.647617] [ 4896]  1003  4896   237234    35088     286      59        0           300 chromium
[88664.647618] [ 4921]  1003  4921   236416    34098     287      61        0           300 chromium
[88664.647619] [ 4961]  1003  4961   239345    37760     293      62        0           300 chromium
[88664.647620] [ 4974]  1003  4974   244681    43793     305      60        0           300 chromium
[88664.647621] [ 5050]  1003  5050   231578    32100     273      49        0           300 chromium
[88664.647622] [ 5064]  1003  5064   247431    43004     313      71        0           300 chromium
[88664.647623] [ 5097]  1003  5097   238312    37053     296      64        0           300 chromium
[88664.647624] [ 5148]  1003  5148   254437    41322     339      84        0           300 chromium
[88664.647625] [ 5161]  1003  5161   237117    35520     293      59        0           300 chromium
[88664.647626] [ 5178]  1003  5178   221792    26662     259      49        0           300 chromium
[88664.647627] [ 5240]  1003  5240   223372    29933     265      50        0           300 chromium
[88664.647628] [ 5257]  1003  5257   222314    26071     261      48        0           300 chromium
[88664.647630] [ 5280]  1003  5280   222314    25925     258      48        0           300 chromium
[88664.647631] [ 5306]  1003  5306   240108    28375     273      44        0           300 chromium
[88664.647632] [ 5332]  1003  5332   222952    24615     257      44        0           300 chromium
[88664.647632] [ 5342]  1003  5342   229004    29209     265      46        0           300 chromium
[88664.647634] [ 5418]  1003  5418   232230    31798     276      51        0           300 chromium
[88664.647636] [ 5468]  1003  5468   215095    25277     254      48        0           300 chromium
[88664.647637] [ 5521]  1003  5521   222014    24540     252      44        0           300 chromium
[88664.647638] [ 5534]  1003  5534   216776    24237     253      48        0           300 chromium
[88664.647639] [ 5543]  1003  5543   306884    83475     496     124        0           300 chromium
[88664.647640] [ 5564]  1003  5564   239317    33578     297      62        0           300 chromium
[88664.647641] [ 5633]  1003  5633   246976    41563     330      84        0           300 chromium
[88664.647642] [ 5674]  1003  5674   250826    48486     329      73        0           300 chromium
[88664.647643] [ 5748]  1003  5748   244379    43492     303      60        0           300 chromium
[88664.647644] [ 5780]  1003  5780   316274   107199     514     163        0           300 chromium
[88664.647645] [ 5807]  1003  5807   267724    56806     365      98        0           300 chromium
[88664.647646] [ 5830]  1003  5830   279745    66927     404     107        0           300 chromium
[88664.647647] [ 5939]  1003  5939   247823    43061     326      73        0           300 chromium
[88664.647648] [ 7326]  1003  7326   164511    12264     203      15        0           300 chromium
[88664.647649] [23214]  1003 23214   243822    38613     292      49        0           300 chromium
[88664.647650] [20076]  1003 20076    86099     1942      36       3        0             0 at-spi-bus-laun
[88664.647652] [20081]  1003 20081     8186      723      21       3        0             0 dbus-daemon
[88664.647654] [20083]  1003 20083    55264     1449      42       3        0             0 at-spi2-registr
[88664.647656] [  691]  1003   691   280810    49025     402     127        0           300 chromium
[88664.647658] [19755]  1003 19755     9779     1965      25       3        0             0 zsh
[88664.647659] [19924]  1003 19924     9779     1932      23       3        0             0 zsh
[88664.647660] [29338]  1003 29338   261044    44880     337      83        0           300 chromium
[88664.647661] [ 6482]  1003  6482   241143    29687     279      50        0           300 chromium
[88664.647663] [29081]  1003 29081     4649      829      13       3        0             0 zsh
[88664.647664] [29115]  1003 29115   242737    22002     163       5        0             0 Main
[88664.647665] [31563]  1003 31563     9485     1312      22       3        0             0 ssh
[88664.647666] [32492]  1003 32492     9485     1353      22       3        0             0 ssh
[88664.647667] [  726]  1003   726     3983       53      13       3        0             0 xsel
[88664.647668] [ 3867]  1003  3867   495708    10455      61       5        0             0 urxvt
[88664.647669] [ 3868]  1003  3868     9778     1945      22       4        0             0 zsh
[88664.647670] [ 3970]  1003  3970     9485     1353      24       3        0             0 ssh
[88664.647672] [ 9907]     0  9907    17510     1306      37       3        0             0 sudo
[88664.647673] [ 9953]     0  9953   102926    14301     205       3        0             0 rsync
[88664.647675] [ 9954]     0  9954    39320     7677      82       3        0             0 rsync
[88664.647676] [ 9955]     0  9955   105278     9764     212       3        0             0 rsync
[88664.647677] Out of memory: Kill process 3756 (chromium) score 329 or sacrifice child
[88664.647701] Killed process 3756 (chromium) total-vm:1933928kB, anon-rss:888836kB, file-rss:73232kB, shmem-rss:1492kB



[88699.560672] Xorg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
[88699.560674] Xorg cpuset=/ mems_allowed=0
[88699.560678] CPU: 2 PID: 953 Comm: Xorg Tainted: P           O    4.8.2-1-ARCH #1
[88699.560678] Hardware name: System manufacturer System Product Name/P8P67 REV 3.1, BIOS 3602 11/01/2012
[88699.560680]  0000000000000286 0000000011eed91e ffff8803f875b978 ffffffff812fe280
[88699.560681]  ffff8803f875bb38 ffff880349c20e40 ffff8803f875b9e8 ffffffff81205f32
[88699.560683]  ffff8803f875b980 0000000000000000 0000000000000000 0000000000000000
[88699.560684] Call Trace:
[88699.560690]  [<ffffffff812fe280>] dump_stack+0x63/0x83
[88699.560692]  [<ffffffff81205f32>] dump_header+0x5c/0x1ea
[88699.560694]  [<ffffffff8117ca25>] oom_kill_process+0x265/0x410
[88699.560696]  [<ffffffff810864c7>] ? has_capability_noaudit+0x17/0x20
[88699.560697]  [<ffffffff8117cfa0>] out_of_memory+0x380/0x420
[88699.560699]  [<ffffffff81313118>] ? find_next_bit+0x18/0x20
[88699.560701]  [<ffffffff81182600>] __alloc_pages_nodemask+0xda0/0xde0
[88699.560704]  [<ffffffff811d6585>] alloc_pages_current+0x95/0x140
[88699.560706]  [<ffffffff811a3e3e>] kmalloc_order_trace+0x2e/0xf0
[88699.560707]  [<ffffffff811e1eca>] __kmalloc+0x1ea/0x200
[88699.560720]  [<ffffffffa17a02c7>] nvkms_alloc+0x27/0x60 [nvidia_modeset]
[88699.560725]  [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88699.560731]  [<ffffffffa17daf2a>] _nv001929kms+0x1a/0x30 [nvidia_modeset]
[88699.560737]  [<ffffffffa17c7282>] ? _nv001878kms+0x32/0xcf0 [nvidia_modeset]
[88699.560738]  [<ffffffff811e1eca>] ? __kmalloc+0x1ea/0x200
[88699.560739]  [<ffffffff81205d54>] ? __check_object_size+0x54/0x1d6
[88699.560744]  [<ffffffffa17a1580>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[88699.560748]  [<ffffffffa17a15ae>] ? _nv000169kms+0x2e/0x40 [nvidia_modeset]
[88699.560752]  [<ffffffffa17a2101>] ? nvKmsIoctl+0x161/0x1e0 [nvidia_modeset]
[88699.560757]  [<ffffffffa17a0da5>] ? nvkms_ioctl_common+0x45/0x80 [nvidia_modeset]
[88699.560761]  [<ffffffffa17a0e51>] ? nvkms_ioctl+0x71/0xa0 [nvidia_modeset]
[88699.560907]  [<ffffffffa0b7a080>] ? nvidia_frontend_compat_ioctl+0x40/0x50 [nvidia]
[88699.560945]  [<ffffffffa0b7a09e>] ? nvidia_frontend_unlocked_ioctl+0xe/0x10 [nvidia]
[88699.560947]  [<ffffffff8121c313>] ? do_vfs_ioctl+0xa3/0x5f0
[88699.560948]  [<ffffffff81088f9b>] ? recalc_sigpending+0x1b/0x50
[88699.560949]  [<ffffffff81089af1>] ? __set_task_blocked+0x41/0xa0
[88699.560950]  [<ffffffff8121c8d9>] ? SyS_ioctl+0x79/0x90
[88699.560951]  [<ffffffff8108c644>] ? SyS_rt_sigprocmask+0x74/0xc0
[88699.560954]  [<ffffffff815f7cf2>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
[88699.560958] Mem-Info:
[88699.560960] active_anon:2556879 inactive_anon:277647 isolated_anon:0
                active_file:496226 inactive_file:211590 isolated_file:64
                unevictable:0 dirty:11020 writeback:10052 unstable:0
                slab_reclaimable:385508 slab_unreclaimable:32365
                mapped:139836 shmem:71185 pagetables:32910 bounce:0
                free:33424 free_pcp:26 free_cma:0
[88699.560963] Node 0 active_anon:10227516kB inactive_anon:1110588kB active_file:1984904kB inactive_file:846360kB unevictable:0kB isolated(anon):0kB isolated(file):256kB mapped:559344kB dirty:44080kB writeback:40208kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 4503552kB anon_thp: 284740kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[88699.560964] Node 0 DMA free:15888kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88699.560966] lowmem_reserve[]: 0 3463 15989 15989 15989
[88699.560968] Node 0 DMA32 free:64652kB min:14620kB low:18272kB high:21924kB active_anon:2212312kB inactive_anon:56780kB active_file:610596kB inactive_file:132264kB unevictable:0kB writepending:29820kB present:3636768kB managed:3561020kB mlocked:0kB slab_reclaimable:444048kB slab_unreclaimable:14604kB kernel_stack:288kB pagetables:4196kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88699.560970] lowmem_reserve[]: 0 0 12526 12526 12526
[88699.560971] Node 0 Normal free:53156kB min:52892kB low:66112kB high:79332kB active_anon:8015204kB inactive_anon:1053808kB active_file:1374392kB inactive_file:714096kB unevictable:0kB writepending:54592kB present:13090816kB managed:12827296kB mlocked:0kB slab_reclaimable:1097984kB slab_unreclaimable:114844kB kernel_stack:21296kB pagetables:127444kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB
[88699.560973] lowmem_reserve[]: 0 0 0 0 0
[88699.560975] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB
[88699.560980] Node 0 DMA32: 5266*4kB (UME) 4744*8kB (UME) 353*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64664kB
[88699.560984] Node 0 Normal: 13104*4kB (UMH) 22*8kB (H) 0*16kB 11*32kB (H) 6*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 53328kB
[88699.560990] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[88699.560990] 779085 total pagecache pages
[88699.560991] 4 pages in swap cache
[88699.560992] Swap cache stats: add 94, delete 90, find 0/0
[88699.560992] Free swap  = 16776844kB
[88699.560992] Total swap = 16777212kB
[88699.560993] 4185892 pages RAM
[88699.560993] 0 pages HighMem/MovableOnly
[88699.560993] 84838 pages reserved
[88699.560993] 0 pages hwpoisoned
[88699.560994] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[88699.561001] [  357]     0   357    38691     8485      68       3        0             0 systemd-journal
[88699.561008] [  465]     0   465     3032      539      11       3        0             0 paxd
[88699.561010] [  474]     0   474     9141      947      19       3        0         -1000 systemd-udevd
[88699.561011] [  647]     0   647    43505      360      18       3        0             0 lvmetad
[88699.561015] [  780]   192   780    28539      970      26       3        0             0 systemd-timesyn
[88699.561016] [  784]    84   784    10327      835      24       3        0             0 avahi-daemon
[88699.561017] [  785]    84   785    10295       77      23       3        0             0 avahi-daemon
[88699.561018] [  792]     0   792     3259      642      12       3        0             0 crond
[88699.561019] [  793]     0   793     9629     1123      25       3        0             0 systemd-logind
[88699.561020] [  796]    81   796     8355      995      21       3        0          -900 dbus-daemon
[88699.561021] [  814]     0   814   108612     3521      74       4        0             0 NetworkManager
[88699.561022] [  821]  1003   821   178158     6659      65       4        0             0 transmission-da
[88699.561023] [  826]     0   826    21924     1682      47       3        0             0 cupsd
[88699.561025] [  829]     0   829    10100      953      25       3        0         -1000 sshd
[88699.561026] [  831]     0   831    22532      945      49       3        0             0 login
[88699.561027] [  833]   124   833    76298     2522      51       4        0             0 colord
[88699.561028] [  838]     0   838    64689     2320      60       4        0             0 cups-browsed
[88699.561029] [  844]   102   844   131172     3821      53       4        0             0 polkitd
[88699.561030] [  880]     0   880     3702     2182      12       3        0             0 dhclient
[88699.561031] [  909]  1003   909    13724     1551      30       3        0             0 systemd
[88699.561032] [  910]  1003   910    24712      461      49       3        0             0 (sd-pam)
[88699.561033] [  917]  1003   917     3406      781      11       3        0             0 startx
[88699.561034] [  952]  1003   952     3979      228      15       3        0             0 xinit
[88699.561035] [  953]  1003   953    61019    14216      93       4        0             0 Xorg
[88699.561036] [  963]  1003   963    32606     3724      66       4        0             0 i3
[88699.561038] [  967]  1003   967     8219      738      21       3        0             0 dbus-daemon
[88699.561039] [  973]  1003   973    15053     1103      32       3        0             0 redshift
[88699.561040] [  979]  1003   979     5029      252      15       3        0             0 xautolock
[88699.561041] [  985]  1003   985    28419     4532      59       4        0             0 compton
[88699.561042] [  987]  1003   987    27077     1967      56       3        0             0 i3bar
[88699.561043] [  990]  1003   990   292303    11908      65       5        0             0 urxvt
[88699.561044] [  992]  1003   992   207220    22365     163       4        0             0 pcmanfm
[88699.561046] [  993]  1003   993   189775     3626      45       4        0             0 python3
[88699.561047] [  997]  1003   997   139110     3981     111       3        0             0 pulseaudio
[88699.561048] [ 1007]  1003  1007    18073     1449      29       3        0             0 xbindkeys
[88699.561049] [ 1008]   133  1008    44462      657      22       4        0             0 rtkit-daemon
[88699.561050] [ 1017]  1003  1017    69992     1508      36       4        0             0 gvfsd
[88699.561051] [ 1022]  1003  1022   103333     1345      36       4        0             0 gvfsd-fuse
[88699.561052] [ 1031]  1003  1031     4649      819      13       3        0             0 zsh
[88699.561054] [ 1035]  1003  1035     4887      714      14       3        0             0 tmux: client
[88699.561054] [ 1039]  1003  1039     7035     2369      18       3        0             0 tmux: server
[88699.561055] [ 1041]  1003  1041     3406      721      11       3        0             0 sh
[88699.561056] [ 1043]  1003  1043     9061     1699      23       3        0             0 zsh
[88699.561058] [ 1171]  1003  1171   166203     2876      62       4        0             0 gvfs-udisks2-vo
[88699.561058] [ 1217]     0  1217   116008     2325      52       4        0             0 udisksd
[88699.561059] [ 1371]  1003  1371    88997     1529      40       3        0             0 gvfsd-trash
[88699.561061] [ 1380]  1003  1380    89537     2046      41       4        0             0 gvfsd-computer
[88699.561062] [ 1387]  1003  1387   107970     1572      45       4        0             0 gvfsd-network
[88699.561063] [ 1393]  1003  1393    93102     1635      49       3        0             0 gvfsd-dnssd
[88699.561064] [ 1455]  1003  1455    49157     3350      94       3        0             0 vim
[88699.561066] [ 1710]  1003  1710     9779     1974      24       3        0             0 zsh
[88699.561068] [ 1742]  1003  1742     1039       17       8       3        0             0 sshtunnel
[88699.561070] [ 1744]  1003  1744     9646     1584      24       3        0             0 ssh
[88699.561072] [ 1807]  1003  1807     9779     1918      23       3        0             0 zsh
[88699.561073] [ 1904]  1003  1904     9518     1331      23       3        0             0 ssh
[88699.561075] [ 1932]  1003  1932     9568      707      22       3        0             0 ssh
[88699.561077] [ 1983]    99  1983     3406      684      13       3        0             0 pacserve
[88699.561078] [ 1984]    99  1984    90540     5776      78       4        0             0 python3
[88699.561080] [ 2013]  1003  2013     9518     1339      22       3        0             0 ssh
[88699.561082] [ 2017]  1003  2017     9546      625      21       3        0             0 ssh
[88699.561084] [ 2263]  1003  2263    10288     2328      24       3        0             0 zsh
[88699.561085] [ 2609]  1003  2609   475468   131371     602       6        0             0 chromium
[88699.561086] [ 2611]  1003  2611     1585      193       8       3        0             0 chrome-sandbox
[88699.561089] [ 2612]  1003  2612   115217    11933     170       4        0             0 chromium
[88699.561090] [ 2618]  1003  2618     1585      183       8       3        0             0 chrome-sandbox
[88699.561092] [ 2619]  1003  2619     7913     1108      22       4        0             0 nacl_helper
[88699.561093] [ 2623]  1003  2623   115217     2880     130       4        0             0 chromium
[88699.561096] [ 2706]  1003  2706   281580   129119     517       5        0           200 chromium
[88699.561097] [ 2712]  1003  2712   116433     6091     148       4        0           200 chromium
[88699.561099] [ 2720]  1003  2720   194399     9303     174      11        0           300 chromium
[88699.561102] [ 2726]  1003  2726   215159    17706     224      29        0           300 chromium
[88699.561104] [ 2742]  1003  2742   279001    64038     377      63        0           300 chromium
[88699.561106] [ 2750]  1003  2750   300123    75558     450     139        0           300 chromium
[88699.561107] [ 2754]  1003  2754   222911    24174     253      44        0           300 chromium
[88699.561108] [ 2762]  1003  2762   307503    99537     444      85        0           300 chromium
[88699.561110] [ 2771]  1003  2771   219930    20678     235      34        0           300 chromium
[88699.561112] [ 2835]  1003  2835   215415    18520     225      29        0           300 chromium
[88699.561114] [ 2839]  1003  2839   215415    17775     224      29        0           300 chromium
[88699.561115] [ 2866]  1003  2866   222592    21000     233      32        0           300 chromium
[88699.561117] [ 2886]  1003  2886   265074    43120     317      67        0           300 chromium
[88699.561118] [ 2896]  1003  2896   235280    31856     265      40        0           300 chromium
[88699.561120] [ 2899]  1003  2899   221167    22566     243      36        0           300 chromium
[88699.561122] [ 2905]  1003  2905   260572    49416     313      64        0           300 chromium
[88699.561124] [ 2911]  1003  2911   227028    28466     250      34        0           300 chromium
[88699.561125] [ 2928]  1003  2928   243407    33841     305      61        0           300 chromium
[88699.561127] [ 2932]  1003  2932   218563    20449     228      30        0           300 chromium
[88699.561129] [ 2943]  1003  2943   218816    21178     236      31        0           300 chromium
[88699.561131] [ 2947]  1003  2947   216752    20447     228      29        0           300 chromium
[88699.561133] [ 2954]  1003  2954   237878    29705     269      44        0           300 chromium
[88699.561135] [ 2958]  1003  2958   286159    77257     465      94        0           300 chromium
[88699.561136] [ 2987]  1003  2987   207988    17274     218      28        0           300 chromium
[88699.561138] [ 2993]  1003  2993   212086    17521     221      29        0           300 chromium
[88699.561140] [ 3004]  1003  3004   214646    19376     228      32        0           300 chromium
[88699.561142] [ 3008]  1003  3008   207988    18075     219      29        0           300 chromium
[88699.561144] [ 3021]  1003  3021   236335    30046     267      52        0           300 chromium
[88699.561146] [ 3042]  1003  3042   240257    34080     277      59        0           300 chromium
[88699.561147] [ 3133]     0  3133    75065     2029      45       3        0             0 upowerd
[88699.561148] [ 3250]  1003  3250   274553    68585     382      70        0           300 chromium
[88699.561150] [ 3292]  1003  3292   508150   186089    1001     296        0           300 chromium
[88699.561151] [ 3328]  1003  3328   388671   164080     832     356        0           300 chromium
[88699.561153] [ 3415]  1003  3415   307696    88364     456     127        0           300 chromium
[88699.561155] [ 3536]  1003  3536   282332    78604     409     107        0           300 chromium
[88699.561157] [ 3546]  1003  3546   305499   101551     511     175        0           300 chromium
[88699.561159] [ 3674]  1003  3674   224280    25328     255      45        0           300 chromium
[88699.561161] [ 3694]  1003  3694   245342    43043     317      74        0           300 chromium
[88699.561163] [ 3747]  1003  3747   219705    24701     253      43        0           300 chromium
[88699.561165] [ 3806]  1003  3806   360885   131169     629     210        0           300 chromium
[88699.561167] [ 3866]  1003  3866   283736    71387     428     120        0           300 chromium
[88699.561169] [ 3875]  1003  3875   499231   201349     782     252        0           300 chromium
[88699.561170] [ 3981]  1003  3981   372987   131031     673     223        0           300 chromium
[88699.561173] [ 4227]  1003  4227   230134    29402     269      51        0           300 chromium
[88699.561175] [ 4291]  1003  4291   288571    79023     435      90        0           300 chromium
[88699.561177] [ 4472]  1003  4472   282333    81144     413     107        0           300 chromium
[88699.561179] [ 4576]  1003  4576   277599    77367     405     116        0           300 chromium
[88699.561181] [ 4694]  1003  4694   232276    31002     274      49        0           300 chromium
[88699.561183] [ 4702]  1003  4702   256330    45063     335      82        0           300 chromium
[88699.561185] [ 4755]  1003  4755   262128    51582     351      83        0           300 chromium
[88699.561186] [ 4773]  1003  4773   222826    25658     254      43        0           300 chromium
[88699.561189] [ 4793]  1003  4793   245184    39989     310      75        0           300 chromium
[88699.561191] [ 4896]  1003  4896   237234    35088     286      59        0           300 chromium
[88699.561194] [ 4921]  1003  4921   236416    34098     287      61        0           300 chromium
[88699.561196] [ 4961]  1003  4961   239345    37760     293      62        0           300 chromium
[88699.561197] [ 4974]  1003  4974   244681    43793     305      60        0           300 chromium
[88699.561199] [ 5050]  1003  5050   231578    32100     273      49        0           300 chromium
[88699.561200] [ 5064]  1003  5064   247431    43004     313      71        0           300 chromium
[88699.561202] [ 5097]  1003  5097   238312    37053     296      64        0           300 chromium
[88699.561204] [ 5148]  1003  5148   254437    41197     339      84        0           300 chromium
[88699.561206] [ 5161]  1003  5161   237117    35520     293      59        0           300 chromium
[88699.561208] [ 5178]  1003  5178   221792    26662     259      49        0           300 chromium
[88699.561210] [ 5240]  1003  5240   223372    29933     265      50        0           300 chromium
[88699.561212] [ 5257]  1003  5257   222314    26071     261      48        0           300 chromium
[88699.561214] [ 5280]  1003  5280   222314    25925     258      48        0           300 chromium
[88699.561217] [ 5306]  1003  5306   240108    28375     273      44        0           300 chromium
[88699.561219] [ 5332]  1003  5332   222952    24615     257      44        0           300 chromium
[88699.561221] [ 5342]  1003  5342   229004    29209     265      46        0           300 chromium
[88699.561224] [ 5418]  1003  5418   232230    31798     276      51        0           300 chromium
[88699.561226] [ 5468]  1003  5468   215095    25277     254      48        0           300 chromium
[88699.561228] [ 5521]  1003  5521   222014    24540     252      44        0           300 chromium
[88699.561230] [ 5534]  1003  5534   216776    24237     253      48        0           300 chromium
[88699.561232] [ 5543]  1003  5543   306884    83528     496     124        0           300 chromium
[88699.561234] [ 5564]  1003  5564   239317    33383     296      61        0           300 chromium
[88699.561237] [ 5633]  1003  5633   246976    41634     330      84        0           300 chromium
[88699.561239] [ 5674]  1003  5674   250826    48486     329      73        0           300 chromium
[88699.561242] [ 5748]  1003  5748   244379    43492     303      60        0           300 chromium
[88699.561245] [ 5780]  1003  5780   316274   107265     514     163        0           300 chromium
[88699.561247] [ 5807]  1003  5807   267686    56227     363      96        0           300 chromium
[88699.561249] [ 5830]  1003  5830   279745    66861     404     107        0           300 chromium
[88699.561251] [ 5939]  1003  5939   247823    43080     326      73        0           300 chromium
[88699.561253] [ 7326]  1003  7326   164511    12264     203      15        0           300 chromium
[88699.561255] [23214]  1003 23214   243822    38613     292      49        0           300 chromium
[88699.561257] [20076]  1003 20076    86099     1942      36       3        0             0 at-spi-bus-laun
[88699.561259] [20081]  1003 20081     8186      723      21       3        0             0 dbus-daemon
[88699.561261] [20083]  1003 20083    55264     1449      42       3        0             0 at-spi2-registr
[88699.561263] [  691]  1003   691   280810    48909     402     127        0           300 chromium
[88699.561266] [19755]  1003 19755     9779     1965      25       3        0             0 zsh
[88699.561270] [19924]  1003 19924     9779     1932      23       3        0             0 zsh
[88699.561272] [29338]  1003 29338   261044    44978     337      83        0           300 chromium
[88699.561275] [ 6482]  1003  6482   241143    29687     279      50        0           300 chromium
[88699.561280] [29081]  1003 29081     4649      829      13       3        0             0 zsh
[88699.561282] [29115]  1003 29115   242737    22002     163       5        0             0 Main
[88699.561285] [31563]  1003 31563     9485     1312      22       3        0             0 ssh
[88699.561286] [32492]  1003 32492     9485     1353      22       3        0             0 ssh
[88699.561288] [  726]  1003   726     3983       53      13       3        0             0 xsel
[88699.561290] [ 3867]  1003  3867   495708    10455      61       5        0             0 urxvt
[88699.561292] [ 3868]  1003  3868     9778     1945      22       4        0             0 zsh
[88699.561295] [ 3970]  1003  3970     9485     1353      24       3        0             0 ssh
[88699.561298] [ 9907]     0  9907    17510     1306      37       3        0             0 sudo
[88699.561301] [ 9953]     0  9953   103054    16330     206       3        0             0 rsync
[88699.561303] [ 9954]     0  9954    39448     8359      82       3        0             0 rsync
[88699.561304] [ 9955]     0  9955   105406    11564     213       3        0             0 rsync
[88699.561306] Out of memory: Kill process 3875 (chromium) score 324 or sacrifice child
[88699.561352] Killed process 3875 (chromium) total-vm:1996924kB, anon-rss:702236kB, file-rss:73416kB, shmem-rss:29744kB

And here it is happening just now on linux-ck 4.8.4.1:

[82892.878181] Xorg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
[82892.878183] Xorg cpuset=/ mems_allowed=0
[82892.878188] CPU: 2 PID: 862 Comm: Xorg Tainted: P           O    4.8.4-1-ck #1
[82892.878189] Hardware name: System manufacturer System Product Name/P8P67 REV 3.1, BIOS 3602 11/01/2012
[82892.878190]  0000000000000286 0000000053b3df50 ffff8803f6adf978 ffffffff812f2af0
[82892.878192]  ffff8803f6adfb38 ffff88032c188000 ffff8803f6adf9e8 ffffffff811e6a91
[82892.878193]  ffffffff810a94e7 ffffffff8184c700 0000000000000246 ffff8803f6adf9c0
[82892.878194] Call Trace:
[82892.878201]  [<ffffffff812f2af0>] dump_stack+0x63/0x83
[82892.878203]  [<ffffffff811e6a91>] dump_header+0x5c/0x1ea
[82892.878205]  [<ffffffff810a94e7>] ? swake_up+0x37/0x40
[82892.878207]  [<ffffffff810c586d>] ? rcu_report_qs_rsp+0x4d/0x80
[82892.878208]  [<ffffffff8115f035>] oom_kill_process+0x265/0x410
[82892.878209]  [<ffffffff810ca214>] ? rcu_read_unlock_special+0x294/0x2d0
[82892.878210]  [<ffffffff8115f5b0>] out_of_memory+0x380/0x420
[82892.878211]  [<ffffffff813077f8>] ? find_next_bit+0x18/0x20
[82892.878213]  [<ffffffff81164b90>] __alloc_pages_nodemask+0xda0/0xde0
[82892.878215]  [<ffffffff811b8295>] alloc_pages_current+0x95/0x140
[82892.878217]  [<ffffffff811860fe>] kmalloc_order_trace+0x2e/0xf0
[82892.878218]  [<ffffffff811c3a1a>] __kmalloc+0x1ea/0x200
[82892.878232]  [<ffffffffa078e297>] nvkms_alloc+0x27/0x60 [nvidia_modeset]
[82892.878237]  [<ffffffffa078f540>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[82892.878243]  [<ffffffffa07c8eea>] _nv001929kms+0x1a/0x30 [nvidia_modeset]
[82892.878249]  [<ffffffffa07b5242>] ? _nv001878kms+0x32/0xcf0 [nvidia_modeset]
[82892.878250]  [<ffffffff811c3a1a>] ? __kmalloc+0x1ea/0x200
[82892.878251]  [<ffffffff811e68b4>] ? __check_object_size+0x54/0x1d5
[82892.878255]  [<ffffffffa078f540>] ? _nv000318kms+0x40/0x40 [nvidia_modeset]
[82892.878260]  [<ffffffffa078f56e>] ? _nv000169kms+0x2e/0x40 [nvidia_modeset]
[82892.878264]  [<ffffffffa07900c1>] ? nvKmsIoctl+0x161/0x1e0 [nvidia_modeset]
[82892.878268]  [<ffffffffa078ed65>] ? nvkms_ioctl_common+0x45/0x80 [nvidia_modeset]
[82892.878273]  [<ffffffffa078ee11>] ? nvkms_ioctl+0x71/0xa0 [nvidia_modeset]
[82892.878358]  [<ffffffffa08f9080>] ? nvidia_frontend_compat_ioctl+0x40/0x50 [nvidia]
[82892.878396]  [<ffffffffa08f909e>] ? nvidia_frontend_unlocked_ioctl+0xe/0x10 [nvidia]
[82892.878397]  [<ffffffff811fce33>] ? do_vfs_ioctl+0xa3/0x5f0
[82892.878398]  [<ffffffff811fd3f9>] ? SyS_ioctl+0x79/0x90
[82892.878400]  [<ffffffff815eb8b2>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
[82892.878411] Mem-Info:
[82892.878414] active_anon:2231054 inactive_anon:273281 isolated_anon:0
                active_file:838912 inactive_file:189143 isolated_file:34
                unevictable:0 dirty:2959 writeback:12020 unstable:0
                slab_reclaimable:42926 slab_unreclaimable:55448
                mapped:159897 shmem:110433 pagetables:31901 bounce:0
                free:33428 free_pcp:87 free_cma:0
[82892.878417] Node 0 active_anon:8924216kB inactive_anon:1093124kB active_file:3355648kB inactive_file:756572kB unevictable:0kB isolated(anon):0kB isolated(file):136kB mapped:639588kB dirty:11836kB writeback:48080kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 2617344kB anon_thp: 441732kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[82892.878417] Node 0 DMA free:15888kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[82892.878421] lowmem_reserve[]: 0 3463 15989 15989 15989
[82892.878422] Node 0 DMA32 free:64536kB min:14620kB low:18272kB high:21924kB active_anon:1834916kB inactive_anon:69556kB active_file:735752kB inactive_file:119064kB unevictable:0kB writepending:11896kB present:3636768kB managed:3561072kB mlocked:0kB slab_reclaimable:48824kB slab_unreclaimable:68748kB kernel_stack:992kB pagetables:5756kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[82892.878425] lowmem_reserve[]: 0 0 12526 12526 12526
[82892.878426] Node 0 Normal free:53288kB min:52892kB low:66112kB high:79332kB active_anon:7089300kB inactive_anon:1023568kB active_file:2619896kB inactive_file:637636kB unevictable:0kB writepending:48024kB present:13090816kB managed:12827312kB mlocked:0kB slab_reclaimable:122880kB slab_unreclaimable:153228kB kernel_stack:31584kB pagetables:121848kB bounce:0kB free_pcp:300kB local_pcp:120kB free_cma:0kB
[82892.878428] lowmem_reserve[]: 0 0 0 0 0
[82892.878429] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB
[82892.878435] Node 0 DMA32: 4259*4kB (UME) 1377*8kB (UME) 2141*16kB (UMEH) 15*32kB (UH) 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 64772kB
[82892.878440] Node 0 Normal: 8298*4kB (MEH) 2004*8kB (UMEH) 141*16kB (UMEH) 23*32kB (UMH) 6*64kB (MH) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52856kB
[82892.878446] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[82892.878446] 1138511 total pagecache pages
[82892.878447] 0 pages in swap cache
[82892.878448] Swap cache stats: add 29, delete 29, find 0/0
[82892.878448] Free swap  = 16777096kB
[82892.878449] Total swap = 16777212kB
[82892.878449] 4185892 pages RAM
[82892.878449] 0 pages HighMem/MovableOnly
[82892.878450] 84821 pages reserved
[82892.878450] 0 pages hwpoisoned
[82892.878450] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[82892.878459] [  367]     0   367    35484     5074      65       3        0             0 systemd-journal
[82892.878461] [  384]     0   384     3032      478      10       3        0             0 paxd
[82892.878463] [  396]     0   396     9183      873      19       3        0         -1000 systemd-udevd
[82892.878465] [  561]     0   561    43505      267      19       3        0             0 lvmetad
[82892.878469] [  690]   192   690    28539      940      26       3        0             0 systemd-timesyn
[82892.878470] [  695]     0   695     9629     1086      21       3        0             0 systemd-logind
[82892.878471] [  696]    81   696     8364      925      21       3        0          -900 dbus-daemon
[82892.878472] [  703]     0   703     3259      652      11       3        0             0 crond
[82892.878474] [  704]    84   704    10327      720      26       3        0             0 avahi-daemon
[82892.878475] [  705]    84   705    10295       76      24       3        0             0 avahi-daemon
[82892.878476] [  711]     0   711    90175     2130      75       4        0             0 NetworkManager
[82892.878477] [  728]     0   728    21924     1242      45       4        0             0 cupsd
[82892.878478] [  731]     0   731    10100      970      24       3        0         -1000 sshd
[82892.878480] [  734]  1003   734   178158     5376      63       4        0             0 transmission-da
[82892.878481] [  737]   124   737    76308     2026      49       3        0             0 colord
[82892.878482] [  742]     0   742    22554      875      47       3        0             0 login
[82892.878484] [  745]     0   745    64689     1503      59       3        0             0 cups-browsed
[82892.878485] [  750]   102   750   131172     2872      51       3        0             0 polkitd
[82892.878487] [  806]     0   806     3702     2110      11       3        0             0 dhclient
[82892.878488] [  818]  1003   818    13726     1364      31       3        0             0 systemd
[82892.878489] [  820]  1003   820    24713      457      49       3        0             0 (sd-pam)
[82892.878490] [  826]  1003   826     3406      782      11       3        0             0 startx
[82892.878492] [  861]  1003   861     3979      218      14       3        0             0 xinit
[82892.878493] [  862]  1003   862    72339    26684     118       3        0             0 Xorg
[82892.878494] [  870]  1003   870    33095     2069      67       3        0             0 i3
[82892.878495] [  874]  1003   874     8246      828      20       3        0             0 dbus-daemon
[82892.878496] [  880]  1003   880    15053      884      32       3        0             0 redshift
[82892.878497] [  886]  1003   886     5029      252      15       3        0             0 xautolock
[82892.878499] [  892]  1003   892    28428     4053      58       4        0             0 compton
[82892.878500] [  899]  1003   899   191670    25064     168       4        0             0 pcmanfm
[82892.878501] [  904]  1003   904   171855     6585     115       4        0             0 pulseaudio
[82892.878503] [  914]  1003   914    18073     1382      27       3        0             0 xbindkeys
[82892.878504] [  915]   133   915    44462      679      22       4        0             0 rtkit-daemon
[82892.878506] [  924]  1003   924    69992      995      36       3        0             0 gvfsd
[82892.878507] [  929]  1003   929   103333     1409      35       3        0             0 gvfsd-fuse
[82892.878508] [ 1068]  1003  1068   166224     1670      60       4        0             0 gvfs-udisks2-vo
[82892.878510] [ 1095]     0  1095   116030     1900      51       4        0             0 udisksd
[82892.878511] [ 1237]  1003  1237     1039       17       8       3        0             0 sshtunnel
[82892.878513] [ 1240]  1003  1240     9738     1646      23       3        0             0 ssh
[82892.878514] [ 1286]  1003  1286    89015     1482      41       3        0             0 gvfsd-trash
[82892.878515] [ 1295]  1003  1295    89537     1019      41       3        0             0 gvfsd-computer
[82892.878516] [ 1301]  1003  1301   107970     1073      43       3        0             0 gvfsd-network
[82892.878518] [ 1307]  1003  1307    93101     1170      47       4        0             0 gvfsd-dnssd
[82892.878519] [ 1469]    99  1469     3406      715      13       3        0             0 pacserve
[82892.878521] [ 1470]    99  1470    90555     4746      77       4        0             0 python3
[82892.878522] [ 1689]  1003  1689     9553      658      22       3        0             0 ssh
[82892.878523] [ 1952]  1003  1952     9543      684      23       3        0             0 ssh
[82892.878525] [ 2986]  1003  2986  1021587   162893     772       7        0             0 chromium
[82892.878526] [ 2996]  1003  2996     1585      185       9       3        0             0 chrome-sandbox
[82892.878527] [ 2997]  1003  2997   115614     4693     174       3        0             0 chromium
[82892.878529] [ 3004]  1003  3004   115614     2544     126       3        0             0 chromium
[82892.878530] [ 3094]  1003  3094   290565    76174     380       5        0           200 chromium
[82892.878532] [ 3098]  1003  3098   353034    19468     257      41        0           300 chromium
[82892.878533] [ 3104]  1003  3104   341767    12691     224      28        0           300 chromium
[82892.878534] [ 3107]  1003  3107   350032    16478     236      34        0           300 chromium
[82892.878536] [ 3110]  1003  3110   357266    20262     393      46        0           300 chromium
[82892.878540] [ 3113]  1003  3113   392215    49743     382      49        0           300 chromium
[82892.878541] [ 3116]  1003  3116   346612    14140     228      31        0           300 chromium
[82892.878542] [ 3119]  1003  3119   356344    16964     237      32        0           300 chromium
[82892.878544] [ 3130]  1003  3130   387860    39041     317      70        0           300 chromium
[82892.878545] [ 3135]  1003  3135   366765    27022     275      42        0           300 chromium
[82892.878546] [ 3139]  1003  3139   372628    46238     312      58        0           300 chromium
[82892.878547] [ 3143]  1003  3143   363473    25922     275      42        0           300 chromium
[82892.878549] [ 3147]  1003  3147   384348    42765     388     130        0           300 chromium
[82892.878550] [ 3151]  1003  3151   351902    15572     236      31        0           300 chromium
[82892.878551] [ 3155]  1003  3155   406886    43203     401     129        0           300 chromium
[82892.878552] [ 3158]  1003  3158   355231    22323     258      39        0           300 chromium
[82892.878553] [ 3163]  1003  3163   348459    16402     235      31        0           300 chromium
[82892.878554] [ 3166]  1003  3166   372618    26599     280      47        0           300 chromium
[82892.878555] [ 3171]  1003  3171   310541     6398     180      12        0           300 chromium
[82892.878557] [ 3175]  1003  3175   320742    12958     222      29        0           300 chromium
[82892.878557] [ 3178]  1003  3178   339175    12650     222      28        0           300 chromium
[82892.878558] [ 3182]  1003  3182   347881    14879     233      32        0           300 chromium
[82892.878559] [ 3187]  1003  3187   341264    12989     224      28        0           300 chromium
[82892.878561] [ 3195]  1003  3195   375319    31495     303      58        0           300 chromium
[82892.878562] [ 3199]  1003  3199   371535    34963     296      76        0           300 chromium
[82892.878563] [ 3203]  1003  3203   370441    24485     267      41        0           300 chromium
[82892.878564] [ 3212]  1003  3212   132571     4960     140       3        0           200 chromium
[82892.878565] [ 3265]  1003  3265   460787    92492     628     200        0           300 chromium
[82892.878566] [ 3269]  1003  3269   368165    34058     319      78        0           300 chromium
[82892.878567] [ 3552]     0  3552    75043     1428      45       4        0             0 upowerd
[82892.878569] [ 4027]  1003  4027   390763    36076     395      61        0           300 chromium
[82892.878570] [ 4157]  1003  4157   382716    33568     352      59        0           300 chromium
[82892.878571] [ 4169]  1003  4169   386368    46484     369     103        0           300 chromium
[82892.878572] [ 4461]  1003  4461   389677    56117     392      84        0           300 chromium
[82892.878573] [ 4855]  1003  4855   373208    33335     307      54        0           300 chromium
[82892.878574] [ 4868]  1003  4868   387365    45880     367      78        0           300 chromium
[82892.878575] [ 4880]  1003  4880   371951    29088     316      51        0           300 chromium
[82892.878577] [ 4945]  1003  4945   478921   121459     643     204        0           300 chromium
[82892.878578] [ 4964]  1003  4964   370792    29137     304      55        0           300 chromium
[82892.878579] [ 5121]  1003  5121   389788    34942     322      65        0           300 chromium
[82892.878581] [ 5537]  1003  5537   381149    30029     310      63        0           300 chromium
[82892.878583] [ 5705]  1003  5705   365445    26076     303      49        0           300 chromium
[82892.878584] [ 5714]  1003  5714   379106    43542     382      95        0           300 chromium
[82892.878585] [ 5804]  1003  5804   394113    38354     412      57        0           300 chromium
[82892.878586] [ 5819]  1003  5819   375900    31106     308      66        0           300 chromium
[82892.878587] [ 5836]  1003  5836   486490   143378     629     103        0           300 chromium
[82892.878589] [ 5849]  1003  5849   379252    32987     314      73        0           300 chromium
[82892.878590] [ 5873]  1003  5873   386554    42809     413     100        0           300 chromium
[82892.878591] [ 5900]  1003  5900   347168    17964     248      44        0           300 chromium
[82892.878592] [ 5927]  1003  5927   359704    22244     268      43        0           300 chromium
[82892.878593] [ 6013]  1003  6013   375234    36920     327      69        0           300 chromium
[82892.878594] [ 7842]  1003  7842    49039     1330      32       3        0             0 gvfsd-metadata
[82892.878596] [30964]  1003 30964    27077     1272      56       3        0             0 i3bar
[82892.878597] [30965]  1003 30965   370002     3063      59       4        0             0 python3
[82892.878598] [12646]  1003 12646    86099     1048      36       4        0             0 at-spi-bus-laun
[82892.878600] [12651]  1003 12651     8186      645      20       3        0             0 dbus-daemon
[82892.878601] [12653]  1003 12653    55297     1084      42       3        0             0 at-spi2-registr
[82892.878602] [25052]  1003 25052    16400     1056      36       3        0             0 gconfd-2
[82892.878603] [ 1137]  1003  1137   291473     9809      65       5        0             0 urxvt
[82892.878605] [ 1147]  1003  1147     9157     1481      23       3        0             0 zsh
[82892.878606] [ 1201]  1003  1201     4887      624      14       3        0             0 tmux: client
[82892.878607] [ 1203]  1003  1203    12229     7235      29       3        0             0 tmux: server
[82892.878608] [ 1204]  1003  1204     3406      720      11       3        0             0 sh
[82892.878609] [ 1577]  1003  1577   531498     5694     156       6        0             0 vim
[82892.878610] [ 2617]  1003  2617     9778     1612      23       3        0             0 zsh
[82892.878611] [ 2712]  1003  2712     9778     1587      24       3        0             0 zsh
[82892.878613] [ 2809]  1003  2809     9485     1317      25       4        0             0 ssh
[82892.878614] [ 2873]  1003  2873     9485     1295      23       3        0             0 ssh
[82892.878615] [11159]  1003 11159     9793     1676      23       3        0             0 zsh
[82892.878616] [12982]  1003 12982     9957     1823      23       3        0             0 zsh
[82892.878617] [13082]  1003 13082     9158     1473      23       3        0             0 zsh
[82892.878619] [13815]  1003 13815     4093     1380      13       3        0             0 htop
[82892.878620] [28960]  1003 28960   496894   158843     782     195        0           300 chromium
[82892.878621] [28964]  1003 28964   461217   109491     660     283        0           300 chromium
[82892.878622] [28972]  1003 28972   384940    31482     312      65        0           300 chromium
[82892.878623] [28976]  1003 28976   387993    30432     307      61        0           300 chromium
[82892.878624] [28980]  1003 28980   388564    50813     405     119        0           300 chromium
[82892.878625] [28984]  1003 28984   405454    59656     455     136        0           300 chromium
[82892.878626] [28992]  1003 28992   402263    62772     451     142        0           300 chromium
[82892.878628] [29003]  1003 29003   385571    39699     344      91        0           300 chromium
[82892.878629] [29009]  1003 29009   380822    41385     376     100        0           300 chromium
[82892.878630] [29013]  1003 29013   377475    37963     330      75        0           300 chromium
[82892.878631] [29017]  1003 29017   371606    29948     316      63        0           300 chromium
[82892.878632] [29021]  1003 29021   381843    38634     344      75        0           300 chromium
[82892.878633] [29025]  1003 29025   378780    36394     326      67        0           300 chromium
[82892.878635] [29029]  1003 29029   366039    29309     296      54        0           300 chromium
[82892.878636] [29033]  1003 29033   393296    51117     412     119        0           300 chromium
[82892.878637] [29037]  1003 29037   526665   116851     668     244        0           300 chromium
[82892.878638] [29041]  1003 29041   373636    33414     305      56        0           300 chromium
[82892.878639] [29045]  1003 29045   381594    44951     348      82        0           300 chromium
[82892.878641] [29049]  1003 29049   355096    20913     260      40        0           300 chromium
[82892.878642] [29053]  1003 29053   388600    47046     357      73        0           300 chromium
[82892.878643] [29057]  1003 29057   365874    27205     299      68        0           300 chromium
[82892.878644] [29061]  1003 29061   381530    33668     353      83        0           300 chromium
[82892.878646] [29065]  1003 29065   376219    32128     330      83        0           300 chromium
[82892.878647] [29069]  1003 29069   377903    37948     314      66        0           300 chromium
[82892.878648] [29077]  1003 29077   361333    24819     278      51        0           300 chromium
[82892.878650] [29081]  1003 29081   356061    18384     253      38        0           300 chromium
[82892.878650] [29089]  1003 29089   354038    17488     250      38        0           300 chromium
[82892.878652] [29098]  1003 29098   364877    29854     294      59        0           300 chromium
[82892.878653] [31687]  1003 31687   287404     9711     200      15        0           300 chromium
[82892.878654] [32664]  1003 32664     9158     1542      23       3        0             0 zsh
[82892.878663] [ 7395]  1003  7395   386732    42833     340      62        0           300 chromium
[82892.878664] [31743]  1003 31743   563140    35290     448       7        0             0 mpv
[82892.878668] [10680]     0 10680    17510     1061      38       3        0             0 sudo
[82892.878669] [10717]     0 10717     9752     1615      23       3        0             0 zsh
[82892.878698] [15960]  1003 15960     4331      569      13       3        0             0 screen
[82892.878699] [15962]  1003 15962     4331      568      12       3        0             0 screen
[82892.878700] [15963]  1003 15963     9158     1535      21       3        0             0 zsh
[82892.878728] [18396]     0 18396    17519     1016      39       3        0             0 sudo
[82892.878729] [18397]  1003 18397     1494      166       7       3        0             0 pv
[82892.878730] [18398]     0 18398    17519     1019      37       3        0             0 sudo
[82892.878731] [18399]     0 18399     8815      646      22       3        0             0 zfs
[82892.878732] [18400]     0 18400     8815      637      21       3        0             0 zfs
[82892.878733] [15247]  1003 15247   372659    35514     335      73        0           300 chromium
[82892.878735] Out of memory: Kill process 28960 (chromium) score 319 or sacrifice child
[82892.878758] Killed process 28960 (chromium) total-vm:1987576kB, anon-rss:429952kB, file-rss:53348kB, shmem-rss:152072kB

The worst thing is, I've seen swap usage go up to 3GB at times under heavy load on 4.8.4.1-ck, so my swap definitely works. (vm.swappiness is set to 10, by the way. changing it to the default 60 does not help)
I guess the bug is still alive? Or is there some horrible issue/misconfiguration on my end? I've never had anything like this happen before 4.7.

Offline

#10 2016-10-25 12:19:22

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: OOM Issue

Don't focus on the swap, OOM killer can be triggered even if there is free swap, it depends more on the RAM available, in any case I'd guess you have 2 ways to solve this problem :

1) trying to cut down the RAM eating processes in order to save RAM  , use the script for that to pin point them. (try understanding if there is a leak or something wrong with them)

2) If you have a free slot just add more RAM and you are done.

I don't think this is a bug as it would be wide spread and oom-killer is triggered only if the RAM is low (talking from experience), even if you see there is free RAM maybe the "next" program tried to eat 500MB and than crashed but was able to trig oom-killer.

However the weird thing is that on both your logs it was Xorg who invoked oom-killer, which is weird as on my list I have different processes using more RAM than Xorg , how much ram it consumes on your installation?
You may find this very informative : https://linux-mm.org/OOM_Killer

Last edited by r0b0t (2016-10-25 12:20:05)

Offline

#11 2016-10-25 13:11:33

bus
Member
Registered: 2014-10-12
Posts: 25

Re: OOM Issue

r0b0t wrote:

Don't focus on the swap, OOM killer can be triggered even if there is free swap, it depends more on the RAM available

But this has never happened before 4.7, when some people started having this issue. None of my habits changed when I upgraded to 4.7 or 4.8. But now my Chromium tabs crash sometimes, while in the past they would have simply been swapped off.

Here's the output of the script:

 Private  +   Shared  =  RAM used       Program

 80.0 KiB +  12.5 KiB =  92.5 KiB       sshtunnel
104.0 KiB +  18.5 KiB = 122.5 KiB       chrome-sandbox
208.0 KiB +  65.0 KiB = 273.0 KiB       xinit
240.0 KiB +  35.5 KiB = 275.5 KiB       xsel
284.0 KiB +  35.0 KiB = 319.0 KiB       paxd
300.0 KiB +  94.5 KiB = 394.5 KiB       xautolock
360.0 KiB + 100.5 KiB = 460.5 KiB       rtkit-daemon
200.0 KiB + 410.5 KiB = 610.5 KiB       tmux: client
608.0 KiB +  41.5 KiB = 649.5 KiB       crond
396.0 KiB + 308.5 KiB = 704.5 KiB       bless
532.0 KiB + 186.5 KiB = 718.5 KiB       redshift
412.0 KiB + 308.5 KiB = 720.5 KiB       pacserve
424.0 KiB + 305.5 KiB = 729.5 KiB       startx
792.0 KiB +  48.5 KiB = 840.5 KiB       lvmetad
516.0 KiB + 334.0 KiB = 850.0 KiB       systemd-timesyncd
328.0 KiB + 631.0 KiB = 959.0 KiB       avahi-daemon (2)
844.0 KiB + 276.5 KiB =   1.1 MiB       at-spi-bus-launcher
852.0 KiB + 319.0 KiB =   1.1 MiB       gvfsd
804.0 KiB + 389.5 KiB =   1.2 MiB       systemd-logind
  1.1 MiB + 242.5 KiB =   1.3 MiB       sshd
  1.0 MiB + 305.0 KiB =   1.3 MiB       at-spi2-registryd
  1.1 MiB + 281.0 KiB =   1.3 MiB       login
  1.0 MiB + 384.0 KiB =   1.4 MiB       gvfsd-dnssd
  1.0 MiB + 401.0 KiB =   1.4 MiB       gvfsd-trash
  1.0 MiB + 432.5 KiB =   1.5 MiB       gvfsd-network
  1.1 MiB + 398.0 KiB =   1.5 MiB       gvfsd-computer
  1.0 MiB + 641.5 KiB =   1.7 MiB       (sd-pam)
  1.4 MiB + 358.0 KiB =   1.8 MiB       sudo
  1.7 MiB + 569.5 KiB =   2.2 MiB       dbus-daemon (3)
  2.2 MiB +  82.5 KiB =   2.3 MiB       systemd-udevd
  1.6 MiB + 793.0 KiB =   2.3 MiB       cups-browsed
  1.8 MiB + 590.0 KiB =   2.4 MiB       cupsd
  2.1 MiB + 422.5 KiB =   2.5 MiB       upowerd
  1.9 MiB + 608.5 KiB =   2.5 MiB       i3bar
  2.2 MiB + 396.5 KiB =   2.6 MiB       tmux: server
  3.2 MiB + 237.5 KiB =   3.4 MiB       gvfsd-fuse
  2.7 MiB + 707.5 KiB =   3.4 MiB       gvfs-udisks2-volume-monitor
  3.5 MiB + 116.0 KiB =   3.6 MiB       xbindkeys
  2.0 MiB +   2.0 MiB =   4.1 MiB       systemd (2)
  2.7 MiB +   1.4 MiB =   4.1 MiB       i3
  4.0 MiB + 606.0 KiB =   4.6 MiB       colord
  4.1 MiB + 439.5 KiB =   4.6 MiB       udisksd
  4.6 MiB +   1.6 MiB =   6.2 MiB       ssh (5)
  6.5 MiB + 685.5 KiB =   7.2 MiB       NetworkManager
  8.0 MiB +  24.0 KiB =   8.1 MiB       dhclient
  9.6 MiB + 856.0 KiB =  10.4 MiB       pulseaudio
 11.4 MiB + 393.0 KiB =  11.8 MiB       polkitd
  9.7 MiB +   2.9 MiB =  12.6 MiB       compton
 16.9 MiB + 585.5 KiB =  17.5 MiB       transmission-daemon
 16.2 MiB +   5.3 MiB =  21.5 MiB       vim (2)
 18.0 MiB +   4.5 MiB =  22.4 MiB       pcmanfm
 22.8 MiB +   2.1 MiB =  24.8 MiB       python3.5 (2)
 35.8 MiB + 849.0 KiB =  36.6 MiB       urxvt
 37.2 MiB +   2.0 MiB =  39.2 MiB       zsh (10)
 42.6 MiB +   5.8 MiB =  48.4 MiB       Xorg
 45.0 MiB +   7.9 MiB =  52.9 MiB       systemd-journald
 79.6 MiB +   2.8 MiB =  82.5 MiB       ipython (2)
 94.6 MiB +   6.7 MiB = 101.2 MiB       Main
157.3 MiB +   6.5 MiB = 163.8 MiB       mpv
  6.2 GiB + 179.0 MiB =   6.4 GiB       chromium (87)
---------------------------------
                          7.1 GiB
=================================

It's only been 5 hours since boot, so Chromium hasn't gone through its usual course of 9GB yet. I'll post again when uptime gets a little closer to the 80000 seconds mark, at which problems seem to arise.

Offline

#12 2016-10-25 19:07:41

Pneumaticat
Member
Registered: 2016-10-25
Posts: 1

Re: OOM Issue

I've also been having OOM issues since 4.7, with Xorg as well -- I've had to stay on an LTS kernel where this issue doesn't occur.

Offline

#13 2016-10-26 03:52:25

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: OOM Issue

Well chromium using 6.4 GB is quite something, especially on a 8GB max installation. Just out of curiosity how many tabs did you open? Have you tried firefox with the same number of tabs?
In any case if you think this is a bug open it and at least you will get a reply by the devels, however to me it sounds like normal sound behaviour from the kernel.

Offline

#14 2016-10-26 05:44:45

bus
Member
Registered: 2014-10-12
Posts: 25

Re: OOM Issue

r0b0t wrote:

Well chromium using 6.4 GB is quite something

No it's not. I have 60-80 tab open (don't judge me) and a few of them are media-heavy like Twitch, which can sometimes eat 1GB apiece. Nothing unusual, lived like that for years. The difference is back then 16GB of RAM + swap was enough, but now my swap usage is at 100KB and stuff is crashing. Somehow it doesn't feel like "normal sound behaviour from the kernel".

Offline

Board footer

Powered by FluxBB