You are not logged in.
Hi, I've been having random hard freezes (only remedy would to wait it out or push the dreaded power button while saying "shhhhhh, it'll be over soon" and weeping). I have looked at the logs and it seems journald has something to do with it, but I don't know what the problem is. I suspected that firefox/aurora globbling too much memory might be the problem, but the freezes occur even without any programs open at all except openbox and sxhkd.
Here's the relevant log which coincides with one of the times the freeze occurred:
Sep 29 12:14:44 Monaco systemd[1]: systemd-journald.service watchdog timeout (limit 1min)!
Sep 29 12:15:00 Monaco systemd-journal[133]: Journal stopped
Sep 29 12:17:27 Monaco systemd-journal[16848]: Permanent journal is using 32.0M (max allowed 20.0M, trying to leave 2.7G free of 6.0G available → current limit 32.0M).
Sep 29 12:17:34 Monaco systemd-journald[133]: Received SIGTERM from PID 1 (systemd).
Sep 29 12:17:36 Monaco systemd[1]: Unit systemd-journald.service entered failed state.
Sep 29 12:17:40 Monaco systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Sep 29 12:17:43 Monaco systemd[1]: Stopping Journal Service...
Sep 29 12:17:43 Monaco systemd[1]: Starting Journal Service...
Sep 29 12:17:43 Monaco systemd[1]: Started Journal Service.
Sep 29 12:17:49 Monaco systemd-journal[16848]: Journal started
Sep 29 12:15:56 Monaco openvpn[11774]: Connection reset, restarting [0]
Sep 29 12:15:56 Monaco openvpn[11774]: SIGUSR1[soft,connection-reset] received, process restarting
Sep 29 12:15:56 Monaco openvpn[11774]: Restart pause, 5 second(s)
Sep 29 12:16:03 Monaco openvpn[11774]: Re-using pre-shared static key
Sep 29 12:25:56 Monaco systemd-journal[16896]: Permanent journal is using 36.0M (max allowed 20.0M, trying to leave 2.7G free of 6.0G available → current limit 36.0M).
Sep 29 12:25:56 Monaco systemd[1]: Starting Journal Service...
Sep 29 12:25:56 Monaco systemd[1]: systemd-journald.service stop-sigterm timed out. Killing.
Sep 29 12:25:56 Monaco systemd[1]: Starting Journal Service...
Sep 29 12:25:56 Monaco systemd[1]: systemd-journald.service: main process exited, code=killed, status=9/KILL
Sep 29 12:25:56 Monaco systemd[1]: Unit systemd-journald.service entered failed state.
Sep 29 12:25:56 Monaco kernel: termite invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Sep 29 12:25:56 Monaco kernel: termite cpuset=/ mems_allowed=0
Sep 29 12:25:56 Monaco kernel: CPU: 1 PID: 16841 Comm: termite Tainted: P O 3.16.3-1-ARCH #1
Sep 29 12:25:56 Monaco kernel: Hardware name: HP-Pavilion GN551AA-ABA m8200n/Nettle2, BIOS 5.16 08/14/2007
Sep 29 12:25:56 Monaco kernel: 0000000000000000 00000000b33a5399 ffff88004697ba08 ffffffff8152b3bc
Sep 29 12:25:56 Monaco kernel: ffff88003ecd5180 ffff88004697ba80 ffffffff8152a89a ffffffff8106e696
Sep 29 12:25:56 Monaco kernel: ffff88004697ba70 ffffffff810d544c 00000000b33a5399 00000000000200da
Sep 29 12:25:56 Monaco kernel: Call Trace:
Sep 29 12:25:56 Monaco kernel: [<ffffffff8152b3bc>] dump_stack+0x4d/0x6f
Sep 29 12:25:56 Monaco kernel: [<ffffffff8152a89a>] dump_header+0xad/0x235
Sep 29 12:25:56 Monaco kernel: [<ffffffff8106e696>] ? put_online_cpus+0x56/0x90
Sep 29 12:25:56 Monaco kernel: [<ffffffff810d544c>] ? rcu_oom_notify+0xdc/0x110
Sep 29 12:25:56 Monaco kernel: [<ffffffff8114fd0c>] oom_kill_process+0x26c/0x3f0
Sep 29 12:25:56 Monaco kernel: [<ffffffff81077b35>] ? has_ns_capability_noaudit+0x35/0x50
Sep 29 12:25:56 Monaco kernel: [<ffffffff81077b67>] ? has_capability_noaudit+0x17/0x20
Sep 29 12:25:56 Monaco kernel: [<ffffffff811505cb>] out_of_memory+0x52b/0x580
Sep 29 12:25:56 Monaco kernel: [<ffffffff8115631d>] __alloc_pages_nodemask+0xbfd/0xc20
Sep 29 12:25:56 Monaco kernel: [<ffffffff8119c04f>] alloc_pages_vma+0xaf/0x1c0
Sep 29 12:25:56 Monaco kernel: [<ffffffff8117acb4>] do_cow_fault+0x64/0x250
Sep 29 12:25:56 Monaco kernel: [<ffffffff8117ccf4>] handle_mm_fault+0x6c4/0x1090
Sep 29 12:25:56 Monaco kernel: [<ffffffff8105c3cf>] __do_page_fault+0x18f/0x600
Sep 29 12:25:56 Monaco kernel: [<ffffffff81184741>] ? mprotect_fixup+0x151/0x260
Sep 29 12:25:56 Monaco kernel: [<ffffffff8105c862>] do_page_fault+0x22/0x30
Sep 29 12:25:56 Monaco kernel: [<ffffffff81532ff8>] page_fault+0x28/0x30
Sep 29 12:25:56 Monaco kernel: Mem-Info:
Sep 29 12:25:56 Monaco kernel: Node 0 DMA per-cpu:
Sep 29 12:25:56 Monaco kernel: CPU 0: hi: 0, btch: 1 usd: 0
Sep 29 12:25:56 Monaco kernel: CPU 1: hi: 0, btch: 1 usd: 0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32 per-cpu:
Sep 29 12:25:56 Monaco kernel: CPU 0: hi: 186, btch: 31 usd: 171
Sep 29 12:25:56 Monaco kernel: CPU 1: hi: 186, btch: 31 usd: 30
Sep 29 12:25:56 Monaco kernel: active_anon:566070 inactive_anon:141732 isolated_anon:0
active_file:784 inactive_file:815 isolated_file:0
unevictable:34 dirty:0 writeback:24 unstable:0
free:14235 slab_reclaimable:5656 slab_unreclaimable:5394
mapped:10704 shmem:8528 pagetables:6076 bounce:0
free_cma:0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA free:12212kB min:232kB low:288kB high:348kB active_anon:1056kB inactive_anon:1108kB active_file:40kB inactive_file:84kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:60kB kernel_stack:32kB pagetables:28kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2455 all_unreclaimable? yes
Sep 29 12:25:56 Monaco kernel: lowmem_reserve[]: 0 2995 2995 2995
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32 free:44728kB min:44820kB low:56024kB high:67228kB active_anon:2263224kB inactive_anon:565820kB active_file:3096kB inactive_file:3176kB unevictable:136kB isolated(anon):0kB isolated(file):0kB present:3128192kB managed:3069416kB mlocked:136kB dirty:0kB writeback:96kB mapped:42816kB shmem:34112kB slab_reclaimable:22568kB slab_unreclaimable:21516kB kernel_stack:4096kB pagetables:24276kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9562 all_unreclaimable? yes
Sep 29 12:25:56 Monaco kernel: lowmem_reserve[]: 0 0 0 0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA: 7*4kB (UE) 7*8kB (UEM) 2*16kB (U) 12*32kB (UEM) 9*64kB (UM) 1*128kB (E) 1*256kB (E) 1*512kB (E) 2*1024kB (EM) 2*2048kB (ER) 1*4096kB (M) = 12212kB
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32: 244*4kB (UEM) 341*8kB (UE) 250*16kB (UEM) 203*32kB (UEM) 237*64kB (UEM) 94*128kB (UEM) 13*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44728kB
Sep 29 12:25:56 Monaco kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 29 12:25:56 Monaco kernel: 10380 total pagecache pages
Sep 29 12:25:56 Monaco kernel: 265 pages in swap cache
Sep 29 12:25:56 Monaco kernel: Swap cache stats: add 548468, delete 548203, find 29936/36511
Sep 29 12:25:56 Monaco kernel: Free swap = 0kB
Sep 29 12:25:56 Monaco kernel: Total swap = 2116244kB
Sep 29 12:25:56 Monaco kernel: 786042 pages RAM
Sep 29 12:25:56 Monaco kernel: 0 pages HighMem/MovableOnly
Sep 29 12:25:56 Monaco kernel: 14694 pages reserved
Sep 29 12:25:56 Monaco kernel: 0 pages hwpoisoned
Sep 29 12:25:56 Monaco kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 29 12:25:56 Monaco kernel: [ 158] 0 158 8069 14 18 152 -1000 systemd-udevd
Sep 29 12:25:56 Monaco kernel: [ 222] 0 222 4270 60 11 110 0 mount.ntfs-3g
Sep 29 12:25:56 Monaco kernel: [ 234] 0 234 3793 43 11 25 0 systemd-logind
Sep 29 12:25:56 Monaco kernel: [ 235] 0 235 18482 0 40 343 0 cupsd
Sep 29 12:25:56 Monaco kernel: [ 236] 81 236 6275 81 17 82 -900 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 238] 0 238 1106 34 8 21 0 dhcpcd
Sep 29 12:25:56 Monaco kernel: [ 249] 0 249 9997 3 24 146 -1000 sshd
Sep 29 12:25:56 Monaco kernel: [ 251] 32 251 7713 16 20 73 0 rpcbind
Sep 29 12:25:56 Monaco kernel: [ 259] 0 259 8726 0 20 92 0 lircd
Sep 29 12:25:56 Monaco kernel: [ 260] 124 260 72948 2 42 590 0 colord
Sep 29 12:25:56 Monaco kernel: [ 283] 0 283 17646 2 39 143 0 login
Sep 29 12:25:56 Monaco kernel: [ 301] 0 301 69839 66 133 493 0 smbd
Sep 29 12:25:56 Monaco kernel: [ 304] 0 304 69970 86 130 473 0 smbd
Sep 29 12:25:56 Monaco kernel: [ 307] 1000 307 6700 0 16 146 0 systemd
Sep 29 12:25:56 Monaco kernel: [ 309] 1000 309 18439 123 36 429 0 (sd-pam)
Sep 29 12:25:56 Monaco kernel: [ 311] 1000 311 10143 2 24 477 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 332] 1000 332 3817 2 12 109 0 startx
Sep 29 12:25:56 Monaco kernel: [ 354] 1000 354 3980 0 13 54 0 xinit
Sep 29 12:25:56 Monaco kernel: [ 355] 1000 355 65132 9680 112 4643 0 Xorg.bin
Sep 29 12:25:56 Monaco kernel: [ 401] 1000 401 51109 1696 99 4190 0 openbox
Sep 29 12:25:56 Monaco kernel: [ 412] 1000 412 158444 522 134 5366 0 mpd
Sep 29 12:25:56 Monaco kernel: [ 414] 1000 414 3160 21 12 25 0 sxhkd
Sep 29 12:25:56 Monaco kernel: [ 415] 1000 415 52991 173 105 3950 0 volumeicon
Sep 29 12:25:56 Monaco kernel: [ 416] 1000 416 47411 109 96 3123 0 tint2
Sep 29 12:25:56 Monaco kernel: [ 424] 1000 424 4507 0 13 70 0 dbus-launch
Sep 29 12:25:56 Monaco kernel: [ 425] 1000 425 6215 56 17 46 0 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 427] 1000 427 83835 0 32 192 0 at-spi-bus-laun
Sep 29 12:25:56 Monaco kernel: [ 431] 1000 431 6189 23 18 57 0 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 434] 1000 434 30771 40 30 148 0 at-spi2-registr
Sep 29 12:25:56 Monaco kernel: [ 496] 1000 496 71998 496 105 3768 0 termite
Sep 29 12:25:56 Monaco kernel: [ 497] 1000 497 10168 2 25 481 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 518] 1000 518 4781 2 15 66 0 tmux
Sep 29 12:25:56 Monaco kernel: [ 520] 1000 520 5365 74 15 71 0 tmux
Sep 29 12:25:56 Monaco kernel: [ 521] 1000 521 10735 2 26 535 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 1087] 1000 1087 5560 2 16 109 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 1093] 1000 1093 36873 0 38 157 0 chromedriver
Sep 29 12:25:56 Monaco kernel: [ 1278] 1000 1278 8763 2 21 84 0 xfconfd
Sep 29 12:25:56 Monaco kernel: [ 1284] 1000 1284 109910 967 139 4237 0 keepassx
Sep 29 12:25:56 Monaco kernel: [ 1288] 1000 1288 11207 35 26 59 0 gconfd-2
Sep 29 12:25:56 Monaco kernel: [ 1444] 1000 1444 44034 2 22 154 0 dconf-service
Sep 29 12:25:56 Monaco kernel: [ 4107] 1000 4107 46937 1858 91 3445 0 weechat
Sep 29 12:25:56 Monaco kernel: [ 7602] 1000 7602 5560 2 16 109 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 7605] 1000 7605 3818 2 13 121 0 bash
Sep 29 12:25:56 Monaco kernel: [ 7615] 1000 7615 3886 2 13 168 0 bash
Sep 29 12:25:56 Monaco kernel: [ 7616] 1000 7616 1873 2 10 21 0 tee
Sep 29 12:25:56 Monaco kernel: [11774] 99 11774 6141 70 17 71 0 openvpn
Sep 29 12:25:56 Monaco kernel: [13718] 1000 13718 5560 2 17 108 0 zsh
Sep 29 12:25:56 Monaco kernel: [13721] 1000 13721 94096 2319 150 5932 0 sonata
Sep 29 12:25:56 Monaco kernel: [13897] 1000 13897 5560 2 16 108 0 zsh
Sep 29 12:25:56 Monaco kernel: [13900] 1000 13900 1217352 653232 2274 376361 0 aurora
Sep 29 12:25:56 Monaco kernel: [14180] 1000 14180 70205 262 102 4007 0 termite
Sep 29 12:25:56 Monaco kernel: [14182] 1000 14182 87917 5172 78 3419 0 livestreamer
Sep 29 12:25:56 Monaco kernel: [14190] 1000 14190 356932 11656 374 6706 0 vlc
Sep 29 12:25:56 Monaco kernel: [16825] 1000 16825 48332 2732 79 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16841] 1000 16841 48332 2065 77 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16851] 1000 16851 30085 118 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16853] 1000 16853 30085 118 53 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16857] 1000 16857 30085 119 53 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16859] 1000 16859 30085 119 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16861] 1000 16861 30085 118 53 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16863] 1000 16863 30085 118 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16865] 1000 16865 30085 119 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16867] 1000 16867 30085 118 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16869] 1000 16869 30085 119 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16882] 0 16882 69839 88 125 471 0 smbd
Sep 29 12:25:56 Monaco kernel: Out of memory: Kill process 13900 (aurora) score 793 or sacrifice child
Sep 29 12:25:56 Monaco kernel: Killed process 14180 (termite) total-vm:280820kB, anon-rss:1048kB, file-rss:0kB
Sep 29 12:25:56 Monaco kernel: termite invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Sep 29 12:25:56 Monaco kernel: termite cpuset=/ mems_allowed=0
Sep 29 12:25:56 Monaco kernel: CPU: 1 PID: 16841 Comm: termite Tainted: P O 3.16.3-1-ARCH #1
Sep 29 12:25:56 Monaco kernel: Hardware name: HP-Pavilion GN551AA-ABA m8200n/Nettle2, BIOS 5.16 08/14/2007
Sep 29 12:25:56 Monaco kernel: 0000000000000000 00000000b33a5399 ffff88004697ba08 ffffffff8152b3bc
Sep 29 12:25:56 Monaco kernel: ffff88003ecd5180 ffff88004697ba80 ffffffff8152a89a ffffffff8106e696
Sep 29 12:25:56 Monaco kernel: ffff88004697ba70 ffffffff810d544c 00000000b33a5399 00000000000200da
Sep 29 12:25:56 Monaco kernel: Call Trace:
Sep 29 12:25:56 Monaco kernel: [<ffffffff8152b3bc>] dump_stack+0x4d/0x6f
Sep 29 12:25:56 Monaco kernel: [<ffffffff8152a89a>] dump_header+0xad/0x235
Sep 29 12:25:56 Monaco kernel: [<ffffffff8106e696>] ? put_online_cpus+0x56/0x90
Sep 29 12:25:56 Monaco kernel: [<ffffffff810d544c>] ? rcu_oom_notify+0xdc/0x110
Sep 29 12:25:56 Monaco kernel: [<ffffffff8114fd0c>] oom_kill_process+0x26c/0x3f0
Sep 29 12:25:56 Monaco kernel: [<ffffffff81077b35>] ? has_ns_capability_noaudit+0x35/0x50
Sep 29 12:25:56 Monaco kernel: [<ffffffff81077b67>] ? has_capability_noaudit+0x17/0x20
Sep 29 12:25:56 Monaco kernel: [<ffffffff811505cb>] out_of_memory+0x52b/0x580
Sep 29 12:25:56 Monaco kernel: [<ffffffff8115631d>] __alloc_pages_nodemask+0xbfd/0xc20
Sep 29 12:25:56 Monaco kernel: [<ffffffff8119c04f>] alloc_pages_vma+0xaf/0x1c0
Sep 29 12:25:56 Monaco kernel: [<ffffffff8117acb4>] do_cow_fault+0x64/0x250
Sep 29 12:25:56 Monaco kernel: [<ffffffff8117ccf4>] handle_mm_fault+0x6c4/0x1090
Sep 29 12:25:56 Monaco kernel: [<ffffffff8105c3cf>] __do_page_fault+0x18f/0x600
Sep 29 12:25:56 Monaco kernel: [<ffffffff81184741>] ? mprotect_fixup+0x151/0x260
Sep 29 12:25:56 Monaco kernel: [<ffffffff8105c862>] do_page_fault+0x22/0x30
Sep 29 12:25:56 Monaco kernel: [<ffffffff81532ff8>] page_fault+0x28/0x30
Sep 29 12:25:56 Monaco kernel: Mem-Info:
Sep 29 12:25:56 Monaco kernel: Node 0 DMA per-cpu:
Sep 29 12:25:56 Monaco kernel: CPU 0: hi: 0, btch: 1 usd: 0
Sep 29 12:25:56 Monaco kernel: CPU 1: hi: 0, btch: 1 usd: 0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32 per-cpu:
Sep 29 12:25:56 Monaco kernel: CPU 0: hi: 186, btch: 31 usd: 176
Sep 29 12:25:56 Monaco kernel: CPU 1: hi: 186, btch: 31 usd: 33
Sep 29 12:25:56 Monaco kernel: active_anon:566070 inactive_anon:141732 isolated_anon:0
active_file:792 inactive_file:793 isolated_file:0
unevictable:34 dirty:0 writeback:24 unstable:0
free:14235 slab_reclaimable:5656 slab_unreclaimable:5394
mapped:10704 shmem:8528 pagetables:6076 bounce:0
free_cma:0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA free:12212kB min:232kB low:288kB high:348kB active_anon:1056kB inactive_anon:1108kB active_file:72kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:60kB kernel_stack:32kB pagetables:28kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2492 all_unreclaimable? yes
Sep 29 12:25:56 Monaco kernel: lowmem_reserve[]: 0 2995 2995 2995
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32 free:44728kB min:44820kB low:56024kB high:67228kB active_anon:2263224kB inactive_anon:565820kB active_file:3096kB inactive_file:3124kB unevictable:136kB isolated(anon):0kB isolated(file):0kB present:3128192kB managed:3069416kB mlocked:136kB dirty:0kB writeback:96kB mapped:42816kB shmem:34112kB slab_reclaimable:22568kB slab_unreclaimable:21516kB kernel_stack:4096kB pagetables:24276kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9658 all_unreclaimable? yes
Sep 29 12:25:56 Monaco kernel: lowmem_reserve[]: 0 0 0 0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA: 7*4kB (UE) 7*8kB (UEM) 2*16kB (U) 12*32kB (UEM) 9*64kB (UM) 1*128kB (E) 1*256kB (E) 1*512kB (E) 2*1024kB (EM) 2*2048kB (ER) 1*4096kB (M) = 12212kB
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32: 244*4kB (UEM) 341*8kB (UE) 250*16kB (UEM) 203*32kB (UEM) 237*64kB (UEM) 94*128kB (UEM) 13*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44728kB
Sep 29 12:25:56 Monaco kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 29 12:25:56 Monaco kernel: 10380 total pagecache pages
Sep 29 12:25:56 Monaco kernel: 265 pages in swap cache
Sep 29 12:25:56 Monaco kernel: Swap cache stats: add 548468, delete 548203, find 29936/36512
Sep 29 12:25:56 Monaco kernel: Free swap = 0kB
Sep 29 12:25:56 Monaco kernel: Total swap = 2116244kB
Sep 29 12:25:56 Monaco kernel: 786042 pages RAM
Sep 29 12:25:56 Monaco kernel: 0 pages HighMem/MovableOnly
Sep 29 12:25:56 Monaco kernel: 14694 pages reserved
Sep 29 12:25:56 Monaco kernel: 0 pages hwpoisoned
Sep 29 12:25:56 Monaco kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 29 12:25:56 Monaco kernel: [ 158] 0 158 8069 14 18 152 -1000 systemd-udevd
Sep 29 12:25:56 Monaco kernel: [ 222] 0 222 4270 60 11 110 0 mount.ntfs-3g
Sep 29 12:25:56 Monaco kernel: [ 234] 0 234 3793 43 11 25 0 systemd-logind
Sep 29 12:25:56 Monaco kernel: [ 235] 0 235 18482 0 40 343 0 cupsd
Sep 29 12:25:56 Monaco kernel: [ 236] 81 236 6275 81 17 82 -900 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 238] 0 238 1106 34 8 21 0 dhcpcd
Sep 29 12:25:56 Monaco kernel: [ 249] 0 249 9997 3 24 146 -1000 sshd
Sep 29 12:25:56 Monaco kernel: [ 251] 32 251 7713 16 20 73 0 rpcbind
Sep 29 12:25:56 Monaco kernel: [ 259] 0 259 8726 0 20 92 0 lircd
Sep 29 12:25:56 Monaco kernel: [ 260] 124 260 72948 2 42 590 0 colord
Sep 29 12:25:56 Monaco kernel: [ 283] 0 283 17646 2 39 143 0 login
Sep 29 12:25:56 Monaco kernel: [ 301] 0 301 69839 66 133 493 0 smbd
Sep 29 12:25:56 Monaco kernel: [ 304] 0 304 69970 86 130 473 0 smbd
Sep 29 12:25:56 Monaco kernel: [ 307] 1000 307 6700 0 16 146 0 systemd
Sep 29 12:25:56 Monaco kernel: [ 309] 1000 309 18439 123 36 429 0 (sd-pam)
Sep 29 12:25:56 Monaco kernel: [ 311] 1000 311 10143 2 24 477 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 332] 1000 332 3817 2 12 109 0 startx
Sep 29 12:25:56 Monaco kernel: [ 354] 1000 354 3980 0 13 54 0 xinit
Sep 29 12:25:56 Monaco kernel: [ 355] 1000 355 65132 9680 112 4643 0 Xorg.bin
Sep 29 12:25:56 Monaco kernel: [ 401] 1000 401 51109 1696 99 4190 0 openbox
Sep 29 12:25:56 Monaco kernel: [ 412] 1000 412 158444 522 134 5366 0 mpd
Sep 29 12:25:56 Monaco kernel: [ 414] 1000 414 3160 21 12 25 0 sxhkd
Sep 29 12:25:56 Monaco kernel: [ 415] 1000 415 52991 173 105 3950 0 volumeicon
Sep 29 12:25:56 Monaco kernel: [ 416] 1000 416 47411 109 96 3123 0 tint2
Sep 29 12:25:56 Monaco kernel: [ 424] 1000 424 4507 0 13 70 0 dbus-launch
Sep 29 12:25:56 Monaco kernel: [ 425] 1000 425 6215 56 17 46 0 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 427] 1000 427 83835 0 32 192 0 at-spi-bus-laun
Sep 29 12:25:56 Monaco kernel: [ 431] 1000 431 6189 23 18 57 0 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 434] 1000 434 30771 40 30 148 0 at-spi2-registr
Sep 29 12:25:56 Monaco kernel: [ 496] 1000 496 71998 496 105 3768 0 termite
Sep 29 12:25:56 Monaco kernel: [ 497] 1000 497 10168 2 25 481 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 518] 1000 518 4781 2 15 66 0 tmux
Sep 29 12:25:56 Monaco kernel: [ 520] 1000 520 5365 74 15 71 0 tmux
Sep 29 12:25:56 Monaco kernel: [ 521] 1000 521 10735 2 26 535 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 1087] 1000 1087 5560 2 16 109 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 1093] 1000 1093 36873 0 38 157 0 chromedriver
Sep 29 12:25:56 Monaco kernel: [ 1278] 1000 1278 8763 2 21 84 0 xfconfd
Sep 29 12:25:56 Monaco kernel: [ 1284] 1000 1284 109910 967 139 4237 0 keepassx
Sep 29 12:25:56 Monaco kernel: [ 1288] 1000 1288 11207 35 26 59 0 gconfd-2
Sep 29 12:25:56 Monaco kernel: [ 1444] 1000 1444 44034 2 22 154 0 dconf-service
Sep 29 12:25:56 Monaco kernel: [ 4107] 1000 4107 46937 1858 91 3445 0 weechat
Sep 29 12:25:56 Monaco kernel: [ 7602] 1000 7602 5560 2 16 109 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 7605] 1000 7605 3818 2 13 121 0 bash
Sep 29 12:25:56 Monaco kernel: [ 7615] 1000 7615 3886 2 13 168 0 bash
Sep 29 12:25:56 Monaco kernel: [ 7616] 1000 7616 1873 2 10 21 0 tee
Sep 29 12:25:56 Monaco kernel: [11774] 99 11774 6141 70 17 71 0 openvpn
Sep 29 12:25:56 Monaco kernel: [13718] 1000 13718 5560 2 17 108 0 zsh
Sep 29 12:25:56 Monaco kernel: [13897] 1000 13897 5560 2 16 108 0 zsh
Sep 29 12:25:56 Monaco kernel: [13900] 1000 13900 1217352 653232 2274 376361 0 aurora
Sep 29 12:25:56 Monaco kernel: [14183] 1000 14180 70205 265 102 4006 0 gmain
Sep 29 12:25:56 Monaco kernel: [14182] 1000 14182 87917 5172 78 3419 0 livestreamer
Sep 29 12:25:56 Monaco kernel: [14190] 1000 14190 356932 11656 374 6706 0 vlc
Sep 29 12:25:56 Monaco kernel: [16825] 1000 16825 48332 2732 79 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16841] 1000 16841 48332 2065 77 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16851] 1000 16851 30085 118 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16853] 1000 16853 30085 118 53 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16857] 1000 16857 30085 119 53 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16859] 1000 16859 30085 119 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16861] 1000 16861 30085 118 53 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16863] 1000 16863 30085 118 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16865] 1000 16865 30085 119 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16867] 1000 16867 30085 118 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16869] 1000 16869 30085 119 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16882] 0 16882 69839 88 125 471 0 smbd
Sep 29 12:25:56 Monaco kernel: Out of memory: Kill process 13900 (aurora) score 793 or sacrifice child
Sep 29 12:25:56 Monaco kernel: Killed process 14183 (gmain) total-vm:280820kB, anon-rss:1052kB, file-rss:8kB
Sep 29 12:25:56 Monaco systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Sep 29 12:25:56 Monaco systemd[1]: Stopping Journal Service...
Sep 29 12:25:56 Monaco systemd[1]: Starting Journal Service...
Sep 29 12:25:56 Monaco systemd[1]: Started Journal Service.
Sep 29 12:25:56 Monaco kernel: termite invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Sep 29 12:25:56 Monaco kernel: termite cpuset=/ mems_allowed=0
Sep 29 12:25:56 Monaco kernel: CPU: 1 PID: 16867 Comm: termite Tainted: P O 3.16.3-1-ARCH #1
Sep 29 12:25:56 Monaco kernel: Hardware name: HP-Pavilion GN551AA-ABA m8200n/Nettle2, BIOS 5.16 08/14/2007
Sep 29 12:25:56 Monaco kernel: 0000000000000000 00000000db03e644 ffff880007f17a08 ffffffff8152b3bc
Sep 29 12:25:56 Monaco kernel: ffff88003ecd5180 ffff880007f17a80 ffffffff8152a89a ffffffff8106e696
Sep 29 12:25:56 Monaco kernel: ffff880007f17a70 ffffffff810d544c 00000000db03e644 00000000000200da
Sep 29 12:25:56 Monaco kernel: Call Trace:
Sep 29 12:25:56 Monaco kernel: [<ffffffff8152b3bc>] dump_stack+0x4d/0x6f
Sep 29 12:25:56 Monaco kernel: [<ffffffff8152a89a>] dump_header+0xad/0x235
Sep 29 12:25:56 Monaco kernel: [<ffffffff8106e696>] ? put_online_cpus+0x56/0x90
Sep 29 12:25:56 Monaco kernel: [<ffffffff810d544c>] ? rcu_oom_notify+0xdc/0x110
Sep 29 12:25:56 Monaco kernel: [<ffffffff8114fd0c>] oom_kill_process+0x26c/0x3f0
Sep 29 12:25:56 Monaco kernel: [<ffffffff81077b35>] ? has_ns_capability_noaudit+0x35/0x50
Sep 29 12:25:56 Monaco kernel: [<ffffffff81077b67>] ? has_capability_noaudit+0x17/0x20
Sep 29 12:25:56 Monaco kernel: [<ffffffff811505cb>] out_of_memory+0x52b/0x580
Sep 29 12:25:56 Monaco kernel: [<ffffffff8115631d>] __alloc_pages_nodemask+0xbfd/0xc20
Sep 29 12:25:56 Monaco kernel: [<ffffffff8119c04f>] alloc_pages_vma+0xaf/0x1c0
Sep 29 12:25:56 Monaco kernel: [<ffffffff8117acb4>] do_cow_fault+0x64/0x250
Sep 29 12:25:56 Monaco kernel: [<ffffffff8117ccf4>] handle_mm_fault+0x6c4/0x1090
Sep 29 12:25:56 Monaco kernel: [<ffffffff8105c3cf>] __do_page_fault+0x18f/0x600
Sep 29 12:25:56 Monaco kernel: [<ffffffff81184741>] ? mprotect_fixup+0x151/0x260
Sep 29 12:25:56 Monaco kernel: [<ffffffff8105c862>] do_page_fault+0x22/0x30
Sep 29 12:25:56 Monaco kernel: [<ffffffff81532ff8>] page_fault+0x28/0x30
Sep 29 12:25:56 Monaco kernel: Mem-Info:
Sep 29 12:25:56 Monaco kernel: Node 0 DMA per-cpu:
Sep 29 12:25:56 Monaco kernel: CPU 0: hi: 0, btch: 1 usd: 0
Sep 29 12:25:56 Monaco kernel: CPU 1: hi: 0, btch: 1 usd: 0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32 per-cpu:
Sep 29 12:25:56 Monaco kernel: CPU 0: hi: 186, btch: 31 usd: 138
Sep 29 12:25:56 Monaco kernel: CPU 1: hi: 186, btch: 31 usd: 128
Sep 29 12:25:56 Monaco kernel: active_anon:565660 inactive_anon:141680 isolated_anon:0
active_file:33 inactive_file:39 isolated_file:0
unevictable:34 dirty:6 writeback:1590 unstable:0
free:16000 slab_reclaimable:5526 slab_unreclaimable:5480
mapped:10239 shmem:8528 pagetables:5958 bounce:0
free_cma:0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA free:12156kB min:232kB low:288kB high:348kB active_anon:1060kB inactive_anon:1208kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:4kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:60kB kernel_stack:32kB pagetables:28kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:68 all_unreclaimable? yes
Sep 29 12:25:56 Monaco kernel: lowmem_reserve[]: 0 2995 2995 2995
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32 free:51844kB min:44820kB low:56024kB high:67228kB active_anon:2261580kB inactive_anon:565512kB active_file:100kB inactive_file:184kB unevictable:136kB isolated(anon):0kB isolated(file):0kB present:3128192kB managed:3069416kB mlocked:136kB dirty:20kB writeback:6360kB mapped:40952kB shmem:34112kB slab_reclaimable:22048kB slab_unreclaimable:21860kB kernel_stack:4224kB pagetables:23804kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:429 all_unreclaimable? yes
Sep 29 12:25:56 Monaco kernel: lowmem_reserve[]: 0 0 0 0
Sep 29 12:25:56 Monaco kernel: Node 0 DMA: 10*4kB (UEM) 5*8kB (UE) 4*16kB (UEM) 10*32kB (UEM) 9*64kB (UM) 1*128kB (E) 1*256kB (E) 1*512kB (E) 2*1024kB (EM) 2*2048kB (ER) 1*4096kB (M) = 12176kB
Sep 29 12:25:56 Monaco kernel: Node 0 DMA32: 1586*4kB (UEM) 519*8kB (UEM) 268*16kB (UEM) 198*32kB (UEM) 227*64kB (UEM) 99*128kB (UEM) 14*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 51904kB
Sep 29 12:25:56 Monaco kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 29 12:25:56 Monaco kernel: 10971 total pagecache pages
Sep 29 12:25:56 Monaco kernel: 2314 pages in swap cache
Sep 29 12:25:56 Monaco kernel: Swap cache stats: add 556759, delete 554445, find 391956/398651
Sep 29 12:25:56 Monaco kernel: Free swap = 0kB
Sep 29 12:25:56 Monaco kernel: Total swap = 2116244kB
Sep 29 12:25:56 Monaco kernel: 786042 pages RAM
Sep 29 12:25:56 Monaco kernel: 0 pages HighMem/MovableOnly
Sep 29 12:25:56 Monaco kernel: 14694 pages reserved
Sep 29 12:25:56 Monaco kernel: 0 pages hwpoisoned
Sep 29 12:25:56 Monaco kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 29 12:25:56 Monaco kernel: [ 158] 0 158 8069 14 18 152 -1000 systemd-udevd
Sep 29 12:25:56 Monaco kernel: [ 222] 0 222 4270 60 11 110 0 mount.ntfs-3g
Sep 29 12:25:56 Monaco kernel: [ 234] 0 234 3793 43 11 25 0 systemd-logind
Sep 29 12:25:56 Monaco kernel: [ 235] 0 235 18482 0 40 343 0 cupsd
Sep 29 12:25:56 Monaco kernel: [ 236] 81 236 6275 81 17 82 -900 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 238] 0 238 1106 34 8 21 0 dhcpcd
Sep 29 12:25:56 Monaco kernel: [ 249] 0 249 9997 3 24 146 -1000 sshd
Sep 29 12:25:56 Monaco kernel: [ 251] 32 251 7713 16 20 73 0 rpcbind
Sep 29 12:25:56 Monaco kernel: [ 259] 0 259 8726 0 20 92 0 lircd
Sep 29 12:25:56 Monaco kernel: [ 260] 124 260 72948 2 42 590 0 colord
Sep 29 12:25:56 Monaco kernel: [ 283] 0 283 17646 2 39 143 0 login
Sep 29 12:25:56 Monaco kernel: [ 301] 0 301 69839 66 133 493 0 smbd
Sep 29 12:25:56 Monaco kernel: [ 304] 0 304 69970 86 130 473 0 smbd
Sep 29 12:25:56 Monaco kernel: [ 307] 1000 307 6700 0 16 146 0 systemd
Sep 29 12:25:56 Monaco kernel: [ 309] 1000 309 18439 123 36 429 0 (sd-pam)
Sep 29 12:25:56 Monaco kernel: [ 311] 1000 311 10143 2 24 477 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 332] 1000 332 3817 2 12 109 0 startx
Sep 29 12:25:56 Monaco kernel: [ 354] 1000 354 3980 0 13 54 0 xinit
Sep 29 12:25:56 Monaco kernel: [ 355] 1000 355 65470 9962 112 4632 0 Xorg.bin
Sep 29 12:25:56 Monaco kernel: [ 401] 1000 401 51109 1696 99 4190 0 openbox
Sep 29 12:25:56 Monaco kernel: [ 412] 1000 412 158444 522 134 5366 0 mpd
Sep 29 12:25:56 Monaco kernel: [ 414] 1000 414 3160 21 12 25 0 sxhkd
Sep 29 12:25:56 Monaco kernel: [ 415] 1000 415 52991 173 105 3950 0 volumeicon
Sep 29 12:25:56 Monaco kernel: [ 416] 1000 416 47411 109 96 3123 0 tint2
Sep 29 12:25:56 Monaco kernel: [ 424] 1000 424 4507 0 13 70 0 dbus-launch
Sep 29 12:25:56 Monaco kernel: [ 425] 1000 425 6215 56 17 46 0 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 427] 1000 427 83835 0 32 192 0 at-spi-bus-laun
Sep 29 12:25:56 Monaco kernel: [ 431] 1000 431 6189 23 18 57 0 dbus-daemon
Sep 29 12:25:56 Monaco kernel: [ 434] 1000 434 30771 40 30 148 0 at-spi2-registr
Sep 29 12:25:56 Monaco kernel: [ 496] 1000 496 71998 499 105 3765 0 termite
Sep 29 12:25:56 Monaco kernel: [ 497] 1000 497 10168 2 25 481 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 518] 1000 518 4781 2 15 66 0 tmux
Sep 29 12:25:56 Monaco kernel: [ 520] 1000 520 5365 74 15 71 0 tmux
Sep 29 12:25:56 Monaco kernel: [ 521] 1000 521 10735 2 26 535 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 1087] 1000 1087 5560 2 16 109 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 1278] 1000 1278 8763 2 21 84 0 xfconfd
Sep 29 12:25:56 Monaco kernel: [ 1284] 1000 1284 109910 967 139 4237 0 keepassx
Sep 29 12:25:56 Monaco kernel: [ 1288] 1000 1288 11207 35 26 59 0 gconfd-2
Sep 29 12:25:56 Monaco kernel: [ 1444] 1000 1444 44034 2 22 154 0 dconf-service
Sep 29 12:25:56 Monaco kernel: [ 4107] 1000 4107 46934 1884 91 3424 0 weechat
Sep 29 12:25:56 Monaco kernel: [ 7602] 1000 7602 5560 2 16 109 0 zsh
Sep 29 12:25:56 Monaco kernel: [ 7605] 1000 7605 3818 2 13 121 0 bash
Sep 29 12:25:56 Monaco kernel: [ 7615] 1000 7615 3886 2 13 168 0 bash
Sep 29 12:25:56 Monaco kernel: [ 7616] 1000 7616 1873 2 10 21 0 tee
Sep 29 12:25:56 Monaco kernel: [11774] 99 11774 6141 70 17 71 0 openvpn
Sep 29 12:25:56 Monaco kernel: [13718] 1000 13718 5560 2 17 108 0 zsh
Sep 29 12:25:56 Monaco kernel: [13897] 1000 13897 5560 2 16 108 0 zsh
Sep 29 12:25:56 Monaco kernel: [13900] 1000 13900 1217352 645627 2274 383966 0 aurora
Sep 29 12:25:56 Monaco kernel: [16825] 1000 16825 48325 2931 90 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16841] 1000 16841 48325 2932 90 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16851] 1000 16851 48332 2316 77 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16853] 1000 16853 48325 2932 87 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16857] 1000 16857 30085 121 53 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16859] 1000 16859 30085 121 54 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16861] 1000 16861 48332 548 69 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16863] 1000 16863 30929 121 55 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16865] 1000 16865 48332 308 68 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16867] 1000 16867 48332 1870 75 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16869] 1000 16869 48332 1864 75 0 0 termite
Sep 29 12:25:56 Monaco kernel: [16896] 0 16896 186 2 3 0 0 systemd-journal
Sep 29 12:25:56 Monaco kernel: [16897] 0 16897 1328 12 4 0 0 systemctl
Sep 29 12:25:56 Monaco kernel: Out of memory: Kill process 13900 (aurora) score 793 or sacrifice child
Sep 29 12:25:56 Monaco kernel: Killed process 13900 (aurora) total-vm:4869408kB, anon-rss:2582508kB, file-rss:0kB
Sep 29 12:25:56 Monaco systemd-journal[16896]: Journal started
Sep 29 12:16:41 Monaco systemd[1]: Starting Trigger Flushing of Journal to Persistent Storage...
Sep 29 12:17:48 Monaco systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
Sep 29 12:17:49 Monaco systemd[1]: Looping too fast. Throttling execution a little.
Sep 29 12:20:19 Monaco systemd[1]: systemd-journald.service watchdog timeout (limit 1min)!
Sep 29 12:25:43 Monaco systemd[1]: Starting Trigger Flushing of Journal to Persistent Storage...
Sep 29 12:25:56 Monaco systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
Sep 29 12:25:57 Monaco systemd[1]: Looping too fast. Throttling execution a little.
I'm also unsure why 12:16:41 timestamp appears *after* 12:25:56!
What I want to know is what might be causing the freezes or how I can figure out what's causing the freezes, and hopefully how to fix it!
Thanks!
edit: I'm aware of the "Out of memory" error related to firefox/aurora, but the freeze happens even without firefox/aurora running!
Last edited by fengshaun (2014-10-09 04:56:51)
Offline
When the freeze happens without aurora running, does the oom killer get invoked at that time, or are there no entries in the log?
Offline
When the freeze happens without aurora running, does the oom killer get invoked at that time, or are there no entries in the log?
Try this: https://wiki.archlinux.org/index.php/Sy … journaling
zʇıɹɟʇıɹʞsuɐs AUR || Cycling in Budapest with a helmet camera || Revised log levels proposal: "FYI" "WTF" and "OMG" (John Barnette)
Offline
Thanks! I had already tried the wiki article about disabling crash dumps and it didn't help! Also, I don't have access to previous logs anymore since "buhman" on #archlinux@irc.freenode.net suggested I remove /var/log/journal directory which solved the problem for now (I waited for a few days before reporting to make sure I don't get freezes).
Also, he had this to say: "feel free to mention that even crap like buhDB is a better implementation of a binary log database"
I'll see when the freezes happen again (or if they do at all) and report back if it happens again.
Thanks!
Last edited by fengshaun (2014-10-12 04:01:42)
Offline