PBS Array Jobs

[Google Code Exporter]

Could you add the capability to submit array jobs with PBS

We need to be able to:
1) Get the variable PBS_ARRAYID on to the command line
2) Dynamically the array size as this will vary from run to run depending on 
(in our case) the number of files

Here is an extract of the python I use to generate the PDS script


    string = """#!/bin/sh
#PBS -N %(job_name)s
#PBS -l %(resources)s
#PBS -t %(array_size)s
cd %(home_dir)s
%(program)s -array_id ${PBS_ARRAYID}
""" % {'job_name': job_name, 'array_size': array_size, 'home_dir': home_dir, 
'program': program, 'resources': resources }

Original issue reported on code.google.com by [e-mail] on 6 Jul 2012 at 8:34

[Google Code Exporter]

We can add arrayid to yabiengine_task. Then for each task that is created for a 
job populate yabiengine_task.arrayid from a 1 based sequence. 
yabiengine_task.arrayid can then be exposed in the pbs template. 

The other component is the array_size. This is best added to yabiengine_job. 
Similarly, we can expose this to the pbs pro template.

Then the above can be achieved (at least in admin) once the jobs, tasks are 
created.

I haven't looked at how to do the task creation for array jobs as yet.

Original comment by [e-mail] on 7 Jul 2012 at 2:55

• Changed title: PBS Array Jobs

[Google Code Exporter]

Original comment by [e-mail] on 7 Jul 2012 at 3:14

• Added labels: Type-Enhancement, Priority-High

[Google Code Exporter]

Would it be possible to get the give some indication of how many of the array 
jobs are complete and how many are still queued?

Original comment by [e-mail] on 9 Jul 2012 at 1:47

[Google Code Exporter]

Original comment by [e-mail] on 3 Sep 2012 at 5:52

• Changed state: Started

[Google Code Exporter]

done in cwellington-1 and merged into next_release rev #ba9d0781b273

New submission template keys are 

tasknum
arrayid

These are synonyms for the same value and are the number of the job in the 
array from 1 to the size of the array

tasktotal
arraysize

These are also the same value and are the total number of jobs in the array.

use mako notation to specify. eg ${arrayid}

Original comment by [e-mail] on 20 Sep 2012 at 8:04

• Changed state: Fixed