[Solved] Long-running R script is getting killed automatically

axelmagn · 2015-11-27 23:13:51

I'm attempting to run an experiment for my masters thesis that involves an R script that has to run for quite a while (up to 24 hours of 100% cpu usage). It's a single-core process, and as far as I can tell I have ample memory for the job. The problem is that when I run it overnight, it's getting killed sometime in the middle of the night without my initiating anything. To give you an idea of the output, here's what I'm looking at:

    ➜  r-lda git:(master) ✗ Rscript slda-test-gibbs-1E6-01.r | tee slda-test-gibbs-1E6-01.log
    Loading required package: lattice
    Loading required package: ggplot2
    Loading required package: methods
    [1] "LOADING DATASET: data/split_1E6.dat"
    [1] "DATASET LOADED"
    [1] "TRAINING DATASET SIZE: 800000"
    [1] "TESTING DATASET SIZE: 200000"
    [1] "VOCAB SIZE: 129276"
    [1] "TRAINING E=50 M=2 K=600"
    [1]    17722 killed     Rscript slda-test-gibbs-1E6-01.r | 
           17723 done       tee slda-test-gibbs-1E6-01.log

I know that's not very informative, but I don't really know how to diagnose this symptom further. It's happened twice now, and I have no idea what's gonig on. I usually don't work much with R scripts, and a bit of googling didn't turn up any real explanation for this. To give you an idea of my system:

    ➜  r-lda git:(master) ✗ uname -a
    Linux ****** 4.2.5-1-ARCH #1 SMP PREEMPT Tue Oct 27 08:13:28 CET 2015 x86_64 GNU/Linux


    ➜  r-lda git:(master) ✗ head -n26 /proc/cpuinfo
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 60
    model name      : Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
    stepping        : 3
    microcode       : 0x12
    cpu MHz         : 3499.863
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 0
    cpu cores       : 4
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt
    bugs            :
    bogomips        : 7015.72
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:

    ➜  r-lda git:(master) ✗ free -h
                  total        used        free      shared  buff/cache   available
    Mem:            15G        3.4G         10G        291M        1.4G         11G
    Swap:           15G        773M         15G

Any insight would be greatly appreciated. I'm sort of pulling my hair out trying to solve this. Thanks in advance.

Last edited by axelmagn (2015-11-28 00:49:12)

WorMzy · 2015-11-27 23:34:33

17722 killed

Something is killing the process. If it isn't you, it's probably the kernel. Check dmesg.

axelmagn · 2015-11-28 00:10:58

You're totally right. It was an OOM error. Apparently the algorithm isn't preallocating the way I thought it was, so the memory would bloat overnight.

jasonwryan · 2015-11-28 00:42:22

Please remember to mark your thread as [Solved] by editing your first post and prepending it to the title.

axelmagn · 2015-11-28 00:49:19

Done

Arch Linux

#1 2015-11-27 23:13:51

[Solved] Long-running R script is getting killed automatically

#2 2015-11-27 23:34:33

Re: [Solved] Long-running R script is getting killed automatically

#3 2015-11-28 00:10:58

Re: [Solved] Long-running R script is getting killed automatically

#4 2015-11-28 00:42:22

Re: [Solved] Long-running R script is getting killed automatically

#5 2015-11-28 00:49:19

Re: [Solved] Long-running R script is getting killed automatically

Board footer