What is this?
You will find two different Python scripts under src directory.
pbs2json.py
pbs2json.py script takes the lines corresponding to finished jobs from Torque's accounting logs and creates a JSON file with specific information about the jobs
A typical Torque accounting log file name looks like this:
YYYYMMDD
A typical finished job line looks like this:
09/12/2016 20:11:56;E;12334.server;user=dilasgoi group=scicomp jobname=JOB2.0194LDFA
queue=p-slow-small ctime=1473500781 qtime=1473500781 etime=1473500781 start=1473506777
owner=dilasgoi@atlas-001 exec_host=cn089/8 Resource_List.cput=200:00:00 Resource_List.mem=1gb
Resource_List.ncpus=1 Resource_List.neednodes=1:ppn=1 Resource_List.nodect=1 Resource_List.nodes=1:ppn=1
session=20136 total_execution_slots=1 unique_node_count=1 end=1473703916 Exit_status=0
resources_used.cput=54:39:13 resources_used.mem=379804kb resources_used.vmem=794444kb
resources_used.walltime=54:45:47
After parsing and reformatting we get something like this:
[
{
"job identifier": "12334",
"username": "dilasgoi",
"group": "scicomp",
"queue": "p-slow-small",
"qtime": "1473500781",
"start": "1473506777",
"end": "1473703916",
"Resource_List.cput": "720000",
"Resource_List.mem": "1",
"total_execution_slots": "1",
"unique_node_count": "1",
"Exit_status": "0",
"resources_used.mem": "379804kb",
"resources_used.walltime": "23342"
}
]
Usage
Usage: pbs2json.py [-h] [-sd SDATE] [-ed EDATE] [-u USER] [-g GROUP]
optional arguments:
-h, --help show this help message and exit
-sd SDATE, --sdate SDATE Start date YYYYMMDD format
-ed EDATE, --edate EDATE End date YYYYMMDD format
-u USER, --user USER Username
-g GROUP, --group GROUP Group
Example:
python pbs2json.py -sd 20160101 -ed 20160102 -u dilasgoi
The script can also create JSON files per user o per group. Providing dates is not mandatory anymore. If no date range is provided the program will scan the whole current year. It also assumes that user belongs to only one group.
pdprocess.py
pdprocess.py script postprocesses JSON files produced by pbs2json.py script with Pandas.
Usage
usage: pdprocess.py [-h] [-f FILE] [-sd SDATE] [-ed EDATE] [-u USER] [-g GROUP]
This program processes JSON files created by pbs2json.py
optional arguments:
-h, --help show this help message and exit
-f FILE, --file FILE File to process
-sd SDATE, --sdate SDATE Start date YYYYMMDD format
-ed EDATE, --edate EDATE End date YYYYMMDD format
-u USER, --user USER Username
-g GROUP, --group GROUP Group
File and start and end dates are not mandatory anymore. If no command line arguments are provided the program expects a file called files/acct_2018.json to exists.