A Simple Job Manager for PBS (PJM)

Summary

PJM is a simple job manager for PBS (Portable Batch System) written by Python. The main function of PJM is inspirated by SJM. Similarlly, PJM can be used as a framework for bioinformatics analysis pipelines by automatically submitting jobs, monitoring job status and controlling the workflow according to dependencies of each job. However, jobs in PJM are defined like "functions", with all the settings including computing resources and job dependencies, which is distinct from SJM.

PJM has beed tested on Torque. But it remains unknown whether PJM also work well on OpenPBS or PBS Professional. PJM may be ported to or compatible with other schedulers in the future, such as SGE (Sun Grid Engine).

Dependencies

Python 2.7

Installation

cd ~ # or anywhere you like
git clone https://github.com/zengxiaofei/PJM.git
export PATH=~/PJM:$PATH # you can add it to .bashrc or .bash_profile

Usage

Print help message:

$ pjm.py -h
usage: pjm.py [-h] [--sleep SLEEP] job_file

** A Simple Job Manager for PBS (PJM) Version 0.9 **
 . Author: Xiaofei Zeng
 . Email:  xiaofei_zeng@whu.edu.cn

positional arguments:
  job_file       input job file

optional arguments:
  -h, --help     show this help message and exit
  --sleep SLEEP  monitor interval time (seconds) [default: 60]

There is only one required argument job_file.

To run a pipeline, just:

$ pjm.py a_pipeline.job

Job definition

A pipeline always contains a series of jobs. Each job has its own demand of resources (memory, processors), and running serially or in parallel in the workflow. It's neccessary and user-friendly to set up these when defining a job.

Here is an example of a job definition:

job_example(depend=another_job;nodes=1;ppn=8;queue=cu;stringency=high;dir=example_dir){
    echo 'This is an example!'
}END

job_example defines the JOBNAME, which is the unique identifier of the job. Avoid duplicated JOBNAME.
depend=jobA means that job_example should be run after another_job finished. Default is no dependency.
nodes=1, ppn=8, queue=cu are settings for the job resources and queue.
stringency=high will add a shell command set -o errexit at the head of generated .pbs file of the job. If an error occurs, the job will be aborted and exit. PJM will wait for other jobs to finish if they are independent with the failed job.
dir=example_dir is the working directory of the job. PJM will creat a folder when this job is ready to run. Default is JOBNAME_dir.
echo 'This is an example!' is a shell command of the job. When you are building a pipeline, the commands can be running softwares, executing custom scripts, or even calling sub-processes.

Configuration

The default settings of nodes, ppn, queue, stringency and avaliable queue list can be edited in configuration file pjm.cfg:

[Default]
nodes=1
ppn=1
queue=cu
stringency=high
[PBS]
QueueList=cu,fat,batch,assemble

zengxiaofei / PJM