Stuff

server_logs/20101006:

10/06/2011 15:52:58;0100;PBS_Server;Job;955.demeter-priv.ccs-priv.edu;enqueuing into ccs_short, state 1 hop 1
10/06/2011 15:52:58;0008;PBS_Server;Job;955.demeter-priv.ccs-priv.edu;Job Queued at request of fuji@sphynx-priv.ccs-priv.edu, owner = fuji@sphynx-priv.ccs-priv.edu, job name = bmt6, queue = ccs_short
10/06/2011 15:55:30;0008;PBS_Server;Job;955.demeter-priv.ccs-priv.edu;Job Run at request of root@demeter-priv.ccs-priv.edu
10/06/2011 15:55:36;0008;PBS_Server;Job;955.demeter-priv.ccs-priv.edu;send of job to compute-02-06.ccs-priv.edu failed error = 15089
10/06/2011 15:55:36;0001;PBS_Server;Svr;PBS_Server;LOG_ERROR::Timed out wating for a reply (15089) in send_job, child failed in previous commit request for job 955.demeter-priv.ccs-priv.edu
10/06/2011 15:55:36;0008;PBS_Server;Job;955.demeter-priv.ccs-priv.edu;unable to run job, MOM rejected/rc=1


maui.log:

10/06 15:55:30 INFO:     192 feasible tasks found for job 955:0 in partition DEFAULT (128 Needed)
10/06 15:55:30 INFO:     tasks located for job 955:  128 of 128 required (128 feasible)
10/06 15:55:30 MJobStart(955)
10/06 15:55:30 MJobDistributeTasks(955,DEMETER-PRIV,NodeList,TaskMap)
10/06 15:55:30 MAMAllocJReserve(955,RIndex,ErrMsg)
10/06 15:55:30 MRMJobStart(955,Msg,SC)
10/06 15:55:30 MPBSJobStart(955,DEMETER-PRIV,Msg,SC)
10/06 15:55:36 ERROR:    job '955' cannot be started: (rc: 15043  errmsg: 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN'  hostlist: 'compute-02-06.ccs-priv.edu:ppn=32+compute-02-05.ccs-priv.edu:ppn=32+compute-02-02.ccs-priv.edu:ppn=32+compute-02-01.ccs-priv.edu:ppn=32')
10/06 15:55:36 ALERT:    cannot start job 955 (RM 'DEMETER-PRIV' failed in function 'jobstart')
10/06 15:55:36 WARNING:  cannot start job '955' through resource manager
10/06 15:55:36 ALERT:    job '955' deferred after 1 failed start attempts (API failure on last attempt)
10/06 15:55:36 MJobSetHold(955,16,00:05:00,RMFailure,cannot start job - RM failure, rc: 15043, msg: 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN')
10/06 15:55:36 ALERT:    job '955' cannot run (deferring job for 300 seconds)
10/06 15:55:36 MSysRegEvent(JOBDEFER:  defer hold placed on job '955'.   reason: 'RMFailure',0,0,1)
10/06 15:55:36 ERROR:    cannot start job '955' in partition DEFAULT
10/06 15:55:36 MJobPReserve(955,DEFAULT,ResCount,ResCountRej)
10/06 15:55:36 MJobReserve(955,Priority)
10/06 15:55:36 INFO:     192 feasible tasks found for job 955:0 in partition DEFAULT (128 Needed)
10/06 15:55:36 INFO:     192 feasible tasks found for job 955:0 in partition DEFAULT (128 Needed)
10/06 15:55:36 INFO:     located resources for 128 tasks (128) in best partition DEFAULT for job 955 at time 00:00:01
10/06 15:55:36 INFO:     tasks located for job 955:  128 of 128 required (128 feasible)
10/06 15:55:36 MJobDistributeTasks(955,DEMETER-PRIV,NodeList,TaskMap)
10/06 15:55:36 MResJCreate(955,MNodeList,00:00:01,Priority,Res)
10/06 15:55:36 INFO:     job '955' reserved 128 tasks (partition DEFAULT) to start in 00:00:01 on Thu Oct  6 15:55:31
10/06 15:55:36 MResDestroy(955)
10/06 15:55:36 MResChargeAllocation(955,2)
10/06 15:55:37 MPBSJobUpdate(955,955.demeter-priv.ccs-priv.edu,TaskList,0)
10/06 15:55:37 INFO:     job '955' Priority:        3
10/06 15:55:37 INFO:     job '955' Priority:        3
10/06 15:55:56 MUGetIndex(checkjob AUTH=root ARG=3 955 0 [NONE] [NONE]
10/06 15:55:56 job '955'  State:        Idle  EState:   Deferred   QueueTime:       Thu Oct  6 15:52:58
10/06 15:56:02 MUGetIndex(checkjob AUTH=root ARG=3 955 0 [NONE] [NONE]
10/06 15:56:02 job '955'  State:        Idle  EState:   Deferred   QueueTime:       Thu Oct  6 15:52:58
10/06 15:56:02 MUGetIndex(checkjob AUTH=root ARG=3 955 0 [NONE] [NONE]
10/06 15:56:02 job '955'  State:        Idle  EState:   Deferred   QueueTime:       Thu Oct  6 15:52:58
10/06 15:56:03 MUGetIndex(checkjob AUTH=root ARG=3 955 0 [NONE] [NONE]
10/06 15:56:03 job '955'  State:        Idle  EState:   Deferred   QueueTime:       Thu Oct  6 15:52:58
10/06 15:56:08 MPBSJobUpdate(955,955.demeter-priv.ccs-priv.edu,TaskList,0)
10/06 15:56:08 INFO:     job '955' Priority:        3
10/06 15:56:08 INFO:     job '955' Priority:        3
10/06 15:56:39 MPBSJobUpdate(955,955.demeter-priv.ccs-priv.edu,TaskList,0)
10/06 15:56:39 INFO:     job '955' Priority:        3
10/06 15:56:39 INFO:     job '955' Priority:        3