Latency problems on HPCs (SLURM, LSF, ...)

[Martin Hölzer]

On HPC systems (e.g. at the EBI LSF cluster) I experienced problems with Nextflow related to latency problems with the file system. For example, NF assumed that .command.sh is available but it took to long to actually write the file to disk and then NF just hangs.

For later reference if we also observe such issues:

EBI slack: I'm now able to run hundreds jobs without slowing down the file system (I keep test it using dd). I believe what made > all the difference was having this in my nextflow.config:

executor {
    name = 'lsf'
    pollInterval = '5 min'
    queueStatInterval = '8 min'
    dumpInterval = '10 min'
}

[Martin Hölzer]

Howto: add the above information as another possible profile latency to the nextflow.config. Then, the user can if needed run something like

-profile slurm,conda,latency

Just skip the name param above.

Profiles can be easily merged so that should work.

In addiion, a note can be added to the README.md in the FAQ or known issues section