Torque resource manager randomly kills process'

jbenge1 · 2018-09-04 03:04:52

I've been attempting to debug this issue for awhile now to no avail. I followed the wiki to set up torque as a resource manager using the built in scheduler pbs_sched.

Here are the mentioned config files (from the wiki) for the head node

/etc/hosts

# Static table lookup for hostnames.
# See hosts(5) for details.
127.0.0.1 localhost
::1       localhost

192.168.1.25 Terra
# Torqe nodes
192.168.1.23 Marte
#192.168.1.120 arch-zenbook

The options specified in the wiki

$ qmgr -c 'p s'
#
# Create queues and set their attributes.
#
#
# Create and define queue batch
#
create queue batch
set queue batch queue_type = Execution
set queue batch resources_default.nodes = 1
set queue batch resources_default.walltime = 01:00:00
set queue batch enabled = True
set queue batch started = True
#
# Set server attributes.
#
set server scheduling = True
set server acl_hosts = Terra
set server managers = justin@Terra
set server operators = justin@Terra
set server default_queue = batch
set server log_events = 511
set server mail_from = adm
set server node_check_rate = 150
set server tcp_timeout = 6
set server job_stat_rate = 300
set server poll_jobs = True
set server down_on_error = True
set server mom_job_sync = True
set server keep_completed = 43200
set server next_job_number = 30
set server moab_array_compatible = True
set server nppcu = 1
set server timeout_for_job_delete = 120
set server timeout_for_job_requeue = 120
set server note_append_on_error = True

 
/var/spool/torque/server_priv/nodes

[sudo] password for justin: 
Terra np=4 gpus=1 Tower
Marte np=4 NFSServer

And the server log for the most recent attempt to run a batch script: https://ptpb.pw/IXNT

The config files for my only compute node (that isn't the head node)

# Static table lookup for hostnames.
# See hosts(5) for details.
#
127.0.0.1 localhost
::1 localhost

# Torque Nodes
192.168.1.25 Terra
192.168.1.24 Giove

/var/spool/torque/mom_priv/config

$pbsserver      Terra
$logevent       225

and the corresponding mom log: https://ptpb.pw/dsAv

The issue that I am having is that whenever I try to run a batch job over the two nodes, it will start however after a random period of time, usually just a few seconds, it is reported as "finished" with no resulting output files in the working directory. After starting a job using

qsub -l nodes=2:ppn=4 /home/user/scripts/someScript

The output of qstat -n is

30.Terra                justin      batch    bubbleSort.py         0     2      8       --   01:00:00 R  00:00:03
   Marte/0-3+Terra/0-3

and a few seconds later is

30.Terra                justin      batch    bubbleSort.py     27808     2      8       --   01:00:00 C       -- 
   Marte/0-3+Terra/0-3

I am at this point at a wall and I don't really know how to proceed. I'd also like to note that I have since removed both nodes Giove and arch-zenbook to reduce the complexity so I could attempt to debug this problem.

Arch Linux

#1 2018-09-04 03:04:52

Torque resource manager randomly kills process'

Board footer