Giters

Nextomics / NextDenovo

Fast and accurate de novo assembler for long reads

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Samtools sort out-of-memory

RenscoHogers opened this issue · comments

commented

Describe the bug
While running NextDenovo on SLURM, I run into a samtools sort error, seemingly due to too little reserved memory.

Error message
Main task:
[1700406 INFO] 2023-05-25 08:28:29 Submitted jobID:[42186471] jobCmd:[/home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align16/nextDenovo.sh] in the slurm_cycle.
[1700406 ERROR] 2023-05-25 08:55:12 ESC[35mctg_align failed: please check the following logs:ESC[0m
[1700406 ERROR] 2023-05-25 08:55:12 ESC[35m/home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01/nextDenovo.sh.eESC[0m

Subtask:
hostname

  • hostname
    cd /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01
  • cd /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01
    time /lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/minimap2-nd --step 3 -x map-ont -a -t 32 /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph1/nd.asm.p.fasta /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/01.raw_align/input.seed.003.2bit|/lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/bam_sort -i -@ 32 -o input.seed.003.2bit.sort.bam
    • time /lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/minimap2-nd/lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/bam_sort --step -i 3 -@ -x 32 map-ont -o -a input.seed.003.2bit.sort.bam -t
      32 /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph1/nd.asm.p.fasta /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/01.raw_align/input.seed.003.2bit
      [M::mm_idx_gen::58.8211.66] collected minimizers
      [M::mm_idx_gen::60.666      2.35] sorted minimizers
      [M::main::60.6692.35] loaded/built the index for 3033 target sequence(s)
      [M::mm_mapopt_update::62.577      2.31] mid_occ = 660
      [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 3033
      [M::mm_idx_stat::63.6932.29] distinct minimizers: 94498812 (41.66% are singletons); average occurrences: 4.866; average spacing: 5.331
      [M::worker_pipeline::158.266      17.91] mapped 22594 sequences
      [M::worker_pipeline::239.393*22.00] mapped 22805 sequences
      Command terminated by signal 9
      5927.23user 178.00system 4:27.79elapsed 2279%CPU (0avgtext+0avgdata 33847516maxresident)k
      5931360inputs+24outputs (0major+85737664minor)pagefaults 0swaps
      [W::sam_read1] Parse error at line 113646
      samtools sort: truncated file. Aborting
      slurmstepd: error: Detected 1 oom-kill event(s) in StepId=42185229.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

Config file
[General]
job_type = slurm
submit = auto
job_prefix = nextDenovo
task = all
rewrite = yes
deltmp = yes
parallel_jobs = 32
input_type = raw
read_type = ont
input_fofn = /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/input.fofn
workdir = /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out

[correct_option]
read_cutoff = 1K
genome_size = 827922386
sort_options = -m 100g -t 32
minimap2_options_raw = -t 32 -m 200g
pa_correction = 8
correction_options = -p 32

[assemble_option]
minimap2_options_cns = -t 32
nextgraph_options = -a 1

Operating system
Which operating system and version are you using?
Ubuntu 20.04 LTS

GCC
What version of GCC are you using?
11.2.0

Python
What version of Python are you using?
3.10.8

NextDenovo
What version of NextDenovo are you using?
2.5.2

commented

see here to set submit. such as sbatch --cpus-per-task={cpu} --mem-per-cpu=2g -o {out} -e {err} {script}

or set submit = sbatch --cpus-per-task={cpu} --mem-per-cpu=2g -o {out} -e {err} {script} in your config file, see details

commented

Hi @RenscoHogers. I'm still trying to figure out how to run NextDenovo in a HPC environment using SLURM. Would you be able to share your script.slurm.sh with me?