Job scheduler (local and on SGE cluster)

SOAP job control(local parallel or SGE parallel).

jobSchedule.generate_job_submit_script(freememory, spname, runtime, nors, parallel=0, mem_total=0, additional_resources='')

Utility function for generate SGE job submission script.

class jobSchedule.localtask(func=None, inputlist=[])

A single local task

monitor()

The task is started on a remote SGE cluster, monitored until all tasks finish, after which afterprocessing will be called.

Some errors will be handled automatically, but others will cause the program to pause, see taskmonitor

Note

It is better to wrap the task in a tasklist object and call tasklist.monitor2end().

jobSchedule.report_job_runstatus(runpath, runsuccess, runnumber, outputname, inputname='runme.py', temppath='')

Utility function for report the job run status when the runs finished.

class jobSchedule.task(dir='', rdirname='', afterprocessing=None, preparing=None, targetfile='')

A single SGE task

Parameters:

• dir (str) – name of the parent directory of the run directory

• rdirname (str) – name of the run directory

• afterprocessing (object) – An ojbect with afterprocessing method for processing the run output, called when the sge run finsihes.

• preparing (object) – An ojbect with prepare_task_input method for prepare the run output, called when the task is started by monitor().

monitor()

The task is started on a remote SGE cluster, monitored until all tasks finish, after which afterprocessing will be called.

Some errors will be handled automatically, but others will cause the program to pause, see taskmonitor

Note

It is better to wrap the task in a tasklist object and call tasklist.monitor2end().

class jobSchedule.taskchain(chains=[])

this calss will chain a list of tasks, and excute the chained functions one by one, including local runs the inididual function should either excute some code, return some result as the input to the next function or the function should result a task object, which can be monitored, and the monitor() return 0 when not success

monitor()

The task is started on a remote SGE cluster, monitored until all tasks finish, after which afterprocessing will be called.

Some errors will be handled automatically, but others will cause the program to pause, see taskmonitor

Note

It is better to wrap the task in a tasklist object and call tasklist.monitor2end().

class jobSchedule.tasklist(tasklist=[], afterprocessing=None, other=None, reload_rundir='', reload_dirlist=[])

List of tasks, task itself can be a jobSchedule.tasklist, jobSchedule.task,:class:jobSchedule.localtask, and jobSchedule.taskchain.

monitor()

The task is started on a remote SGE cluster, monitored until all tasks finish, after which afterprocessing will be called.

Some errors will be handled automatically, but others will cause the program to pause, see taskmonitor

Note

It is better to wrap the task in a tasklist object and call tasklist.monitor2end().

monitor2end()

The list of tasks are started(at the same time), local or on remote SGE cluster, monitored until all tasks finish, after whcih the afterprocessing will be called.

Note

A task, tasklist, or taskchain and be interrupted by CTRL+c at runtime, to perform a list job controlling actions:

@@@@@@@@ task-monitor

You have 120s to interrupt the code to handle the error, otherwise code will continue looping
 0-quit and delete;
 1-quit this job;
 2-re-raise;
 3-quit whole script and delete;
 4-enter debug mode;
 5-continue
 6-restart runs with updated SOAP code
 7-re-parepare and restart job
 8-ignore erros

The task will continue if no input is entered.

The “task-monitor” is also triggered by some fetal errors.