Giters

Nextomics / NextPolish

Fast and accurately polish the genome generated by long reads.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

db_split errors

k-williamson opened this issue · comments

commented

Describe the bug
Hi, I am getting an error during when attempting to run NextPolish, and when I check the log file it indicates, it tells me a variety of files cannot be found. I have the genome and config file in the same directory (NEXTPOLISH) when I start running the program, but once it starts running, it seems to rename this directory to NEXTPOLISH.backup0 and use the old directory name to run the scripts etc. However, based on the log files below, it seems to be looking for the genome and config file in the NEXTPOLISH directory still, even though it has moved them to NEXTPOLISH.backup0. I am not sure why it is doing this or how to fix this, and I am hoping you can advise.

Error message
Main task log:
[39497 INFO] 2022-04-26 11:05:25 NextPolish start...
[39497 INFO] 2022-04-26 11:05:25 version:v1.4.0 logfile:pid39497.log.info
[39497 WARNING] 2022-04-26 11:05:25 Delete task: 5 due to missing lgs_fofn.
[39497 WARNING] 2022-04-26 11:05:25 Delete task: 5 due to missing lgs_fofn.
[39497 WARNING] 2022-04-26 11:05:25 Delete task: 6 due to missing hifi_fofn.
[39497 WARNING] 2022-04-26 11:05:25 Delete task: 6 due to missing hifi_fofn.
[39497 INFO] 2022-04-26 11:05:25 scheduled tasks:
[1, 2, 1, 2]
[39497 INFO] 2022-04-26 11:05:25 options:
[39497 INFO] 2022-04-26 11:05:25
rerun: 3
rewrite: 0
kill: None
cleantmp: 0
use_drmaa: 0
submit: None
job_type: local
sgs_unpaired: 0
sgs_rm_nread: 1
lgs_read_type:
parallel_jobs: 6
align_threads: 5
check_alive: None
task: [1, 2, 1, 2]
job_id_regex: None
sgs_max_depth: 100
lgs_max_depth: 100
multithread_jobs: 5
lgs_max_read_len: 0
hifi_max_depth: 100
lgs_block_size: 500M
lgs_min_read_len: 1k
hifi_max_read_len: 0
genome_size: 30468886
polish_options: -p 5
hifi_block_size: 500M
hifi_min_read_len: 1k
job_prefix: nextPolish
sgs_block_size: 500000000
sgs_use_duplicate_reads: 0
lgs_minimap2_options: -x map-ont
hifi_minimap2_options: -x map-pb
sgs_align_options: bwa mem -p -t 5
workdir: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH
sgs_fofn: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/sgs.fofn
snp_phase: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/%02d.snp_phase
snp_valid: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/%02d.snp_valid
lgs_polish: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/%02d.lgs_polish
kmer_count: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/%02d.kmer_count
hifi_polish: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/%02d.hifi_polish
score_chain: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/%02d.score_chain
genome: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/meteora_purged.medaka.2.fa
[39497 WARNING] 2022-04-26 11:05:25 mv /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH to /misc/scratch4/kelsey/meteora_genome/MinION/POLIS
HING/PURGED_ASSEMBLY/NEXTPOLISH.backup0
[39497 INFO] 2022-04-26 11:05:25 step 0 and task 1 start:
[39497 INFO] 2022-04-26 11:05:30 Total jobs: 3
[39497 INFO] 2022-04-26 11:05:30 Submitted jobID:[39514] jobCmd:[/misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh] in the local_cycle.
[39514 CRITICAL] 2022-04-26 11:05:30 Command '/bin/sh /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh > /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh.o 2> /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh.e' returned non-zero exit status 1, error info: .
[39497 INFO] 2022-04-26 11:05:31 Submitted jobID:[39519] jobCmd:[/misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh] in the local_cycle.
[39519 CRITICAL] 2022-04-26 11:05:31 Command '/bin/sh /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh > /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh.o 2> /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh.e' returned non-zero exit status 1, error info: .
[39497 INFO] 2022-04-26 11:05:31 Submitted jobID:[39524] jobCmd:[/misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split3/nextPolish.sh] in the local_cycle.
[39524 CRITICAL] 2022-04-26 11:05:31 Command '/bin/sh /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split3/nextPolish.sh > /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split3/nextPolish.sh.o 2> /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split3/nextPolish.sh.e' returned non-zero exit status 1, error info: .
[39497 ERROR] 2022-04-26 11:05:38 db_split failed: please check the following logs:
[39497 ERROR] 2022-04-26 11:05:38 /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh.e
[39497 ERROR] 2022-04-26 11:05:38 /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh.e
[39497 ERROR] 2022-04-26 11:05:38 /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split3/nextPolish.sh.e

Failed subtask logs:
hostname

  • hostname
    cd /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split1
  • cd /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split1
    time /misc/scratch2/software/anaconda/envs/nextpolish/custom/NextPolish/bin/seq_split -d /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH -m
    500000000 -n 6 -t 5 -i 1 -s 3046888600 -p input.sgspart /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/sgs.fofn
  • /misc/scratch2/software/anaconda/envs/nextpolish/custom/NextPolish/bin/seq_split -d /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH -m 50
    0000000 -n 6 -t 5 -i 1 -s 3046888600 -p input.sgspart /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/sgs.fofn
    Failed open input file list: /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/sgs.fofn

real 0m0.003s
user 0m0.002s
sys 0m0.000s

hostname

  • hostname
    cd /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split2
  • cd /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split2
    time /misc/scratch2/software/anaconda/envs/nextpolish/custom/NextPolish/bin/bwa index -p /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00
    .score_chain/input.genome.fasta.sgs /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/input.genome.fasta
  • /misc/scratch2/software/anaconda/envs/nextpolish/custom/NextPolish/bin/bwa index -p /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.sc
    ore_chain/input.genome.fasta.sgs /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/input.genome.fasta
    [bwa_idx_build] fail to open file '/misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/input.genome.fasta' : No such file or dir
    ectory

real 0m0.002s
user 0m0.000s
sys 0m0.001s

hostname

  • hostname
    cd /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split3
  • cd /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/01.db_split.sh.work/db_split3
    time /misc/scratch2/software/anaconda/envs/nextpolish/custom/NextPolish/bin/samtools faidx /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/
    00.score_chain/input.genome.fasta
  • /misc/scratch2/software/anaconda/envs/nextpolish/custom/NextPolish/bin/samtools faidx /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.
    score_chain/input.genome.fasta
    [E::fai_build3_core] Failed to open the file /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/input.genome.fasta
    [faidx] Could not build fai index /misc/scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH/00.score_chain/input.genome.fasta.fai

real 0m0.004s
user 0m0.000s
sys 0m0.003s

Operating system
Distributor ID: Ubuntu
Description: Ubuntu 18.04 LTS
Release: 18.04
Codename: bionic

GCC
gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)

Python
Python 3.10.4

NextPolish
nextPolish v1.4.0

Additional info
I am using a conda environment to run this, if that is helpful/relevant
Directory listings after running:
(nextpolish) /scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH.backup0@perun16> ll
total 21M
-rw-r--r-- 1 kelsey roger 30M Apr 26 10:08 meteora_purged.medaka.2.fa
-rw-r--r-- 1 kelsey roger 541 Apr 26 10:12 nextpolish.sh
-rw-r--r-- 1 kelsey roger 194 Apr 26 10:12 sgs.fofn
-rw-r--r-- 1 kelsey roger 68 Apr 26 10:12 run.cfg
-rw-r--r-- 1 kelsey roger 6.0K Apr 26 11:05 pid39497.log.info
(nextpolish) /scratch4/kelsey/meteora_genome/MinION/POLISHING/PURGED_ASSEMBLY/NEXTPOLISH@perun16> ll
total 24K
drwxr-xr-x 3 kelsey roger 5 Apr 26 11:05 00.score_chain

commented

check the rewrite option, if you set rewrite == no, nextPolish will bakcup the working directory to NEXTPOLISH.backup%d if it already existed, you also can change the working directory by the workdir option.