[Question] force SLURM scheduler to submit more jobs

[sirusb]

Hello,
First, thank you very much for the very useful tool, I am a big fan of it.

I am using SLURM scheduler, to map some sequencing data using bowtie, so I configured each bwa job to use up-to 20Gb of RAM. I noticed that bpipe will submit only two jobs and waits until one of them finished. When I give 8Gb to bwa I notice that 5 jobs are launched.

It seems that there is a 40Gb threshold set by bpipe. Is there a way to change that. I couldn't figure how.

My bpipe.config is as follow

executor="slurm"
walltime="0-02:00:00"
procs=8
queue="short"
memory="8g"
modules="gcc/6.2.0 boost samtools/1.3.1 fastqc/0.11.5 bwa/0.7.15 java/jdk-1.8u112"
custom_submit_options="--mail-user=[e-mail] --mail-type=All"
jobLaunchSeparationMs=1000

commands {
    fastqc {
        walltime="0-00:45:00"
    }

    bwa_map {
        walltime="0-09:00:00"
        memory="20g"
        procs=16
    }

    trimmomatic {
        walltime="0-04:00:00"
        memory="16g"
        procs=10
    }

    sam_sort {
        walltime="0-04:00:00"
        memory="20g"
        procs=12
    }
}

Thanks in advance,

[Simon Sadedin]

It might not be a memory limit but a concurrency limit.

Try putting into your bpipe.config

concurrency=200

Bpipe by default limits concurrency to not exceed the amount of processors on the machine it is running on. This is kind of meant to be protection against destroying the host it's running on (which really annoys cluster administrators!). While it doesn't make much sense on a cluster where the jobs don't run on the host that bpipe executes on, I've had enough "accidents" that I have ended up leaving this default in place, so that bpipe not finding its config file for some reason doesn't result in armageddon on the cluster login node.

You can also set at launch time with

bpipe run -n 200 ...

(of course, 200 here is just a number I am picking, you can make it as high as you like).

[sirusb]

Hi Simon,
Thanks for the quick replay.

It makes sense, as the submission node have 32 processors and my bwa are taking 16 processor each. Probably it calculated the limit based on the submission node.