Snakemake fails at rule circularize_final
brymerr921 opened this issue · comments
When providing Illumina and Nanopore reads, the pipeline fails at rule circularize_final
because there is no directory called {sample}/3.circularization/3.circular_sequences/sh
, which should have several files inside. Therefore, it fails with this error message:
519 of 524 steps (99%) done
[Thu Jun 27 23:23:58 2019]
localrule circularize_final:
input: Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa, Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai
output: Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta
jobid: 528
wildcards: sample=Prevotella-stercorea-2632
cat Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/* | bash
ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | cut -f1 -d '_' > circs.tmp || true
(cat Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | grep -vf circs.tmp |
cut -f1 | xargs samtools faidx Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa; ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | xargs cat) |
sed 's/\([ACTG]\)\n//g' | fold -w 120 | cut -f1 -d ':' > Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta || true
#rm circs.tmp
Activating singularity image /mnt/.snakemake/singularity/f075a3ce7bc1893be5302161e5e7807a.simg
Singularity: action-suid (U=0,P=30414)> Not mounting requested bind point (already mounted in container): /mnt/
WARNING: Not mounting requested bind point (already mounted in container): /mnt/
Singularity: action-suid (U=0,P=30414)> Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
WARNING: Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
Singularity: action-suid (U=0,P=30414)> Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
WARNING: Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
Singularity: action-suid (U=0,P=30414)> Skipping user bind, could not create bind point /mnt/snakemake: No such file or directory
WARNING: Skipping user bind, could not create bind point /mnt/snakemake: No such file or directory
Singularity: action-suid (U=0,P=30414)> USER=root, IMAGE='f075a3ce7bc1893be5302161e5e7807a.simg', COMMAND='exec'
cat: 'Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/*': No such file or directory
[Thu Jun 27 23:23:58 2019]
Error in rule circularize_final:
jobid: 528
output: Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta
shell:
cat Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/* | bash
ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | cut -f1 -d '_' > circs.tmp || true
(cat Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | grep -vf circs.tmp |
cut -f1 | xargs samtools faidx Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa; ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | xargs cat) |
sed 's/\([ACTG]\)\n//g' | fold -w 120 | cut -f1 -d ':' > Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta || true
#rm circs.tmp
(exited with non-zero exit code)
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /mnt/.snakemake/log/2019-06-27T231855.973555.snakemake.log
This is the current directory structure:
Prevotella-stercorea-2632/
├── 0.basecall
│ ├── Prevotella-stercorea-2632.fq
│ ├── data_links
│ │ └── dummy
│ └── raw_calls
│ └── dummy
├── 1.assemble
│ ├── Prevotella-stercorea-2632_nomerge.fasta -> /mnt/Prevotella-stercorea-2632/1.assemble/assemble_4m/Prevotella-stercorea-2632_4m.contigs.corrected.fasta
│ ├── Prevotella-stercorea-2632_raw_assembly.fa
│ ├── Prevotella-stercorea-2632_raw_assembly.fa.amb
│ ├── Prevotella-stercorea-2632_raw_assembly.fa.ann
│ ├── Prevotella-stercorea-2632_raw_assembly.fa.bwt
│ ├── Prevotella-stercorea-2632_raw_assembly.fa.fai
│ ├── Prevotella-stercorea-2632_raw_assembly.fa.pac
│ ├── Prevotella-stercorea-2632_raw_assembly.fa.sa
│ └── assemble_4m
│ ├── Prevotella-stercorea-2632_4m.contigs.corrected.fasta
│ ├── Prevotella-stercorea-2632_4m.contigs.fasta
│ ├── Prevotella-stercorea-2632_4m.contigs.fasta.bam
│ ├── Prevotella-stercorea-2632_4m.contigs.fasta.bam.bai
│ ├── Prevotella-stercorea-2632_4m.contigs.fasta.fai
│ ├── Prevotella-stercorea-2632_4m.contigs.fasta.misassemblies.tsv
│ ├── Prevotella-stercorea-2632_4m.contigs.layout
│ ├── Prevotella-stercorea-2632_4m.contigs.layout.readToTig
│ ├── Prevotella-stercorea-2632_4m.contigs.layout.tigInfo
│ ├── Prevotella-stercorea-2632_4m.correctedReads.fasta.gz
│ ├── Prevotella-stercorea-2632_4m.report
│ ├── Prevotella-stercorea-2632_4m.seqStore
│ ├── Prevotella-stercorea-2632_4m.seqStore.err
│ ├── Prevotella-stercorea-2632_4m.seqStore.ssi
│ ├── Prevotella-stercorea-2632_4m.trimmedReads.fasta.gz
│ ├── Prevotella-stercorea-2632_4m.unassembled.fasta
│ ├── Prevotella-stercorea-2632_4m.unitigs.bed
│ ├── Prevotella-stercorea-2632_4m.unitigs.fasta
│ ├── Prevotella-stercorea-2632_4m.unitigs.gfa
│ ├── Prevotella-stercorea-2632_4m.unitigs.layout
│ ├── Prevotella-stercorea-2632_4m.unitigs.layout.readToTig
│ ├── Prevotella-stercorea-2632_4m.unitigs.layout.tigInfo
│ ├── canu-logs
│ ├── canu-scripts
│ ├── correction
│ ├── trimming
│ └── unitigging
├── 2.polish
│ ├── Prevotella-stercorea-2632_polished.fa
│ ├── Prevotella-stercorea-2632_polished.fa.fai
│ └── pilon
│ ├── Prevotella-stercorea-2632_pilon.fa
│ ├── corrections.vcf.gz
│ ├── corrections.vcf.gz.tbi
│ ├── ranges
│ ├── short_reads.bam
│ ├── short_reads.bam.bai
│ └── sub_runs
└── 3.circularization
└── 1.candidate_genomes
Full log file attached:
190627__circularize_final__error.log
I think I see what's happening. Would you mind showing me the result of sort -k2,2gr Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | head
please? I think you've found an edge case that I have failed to account for.
This is the output:
tig00000012 85799 743756 60 61
tig00000007 65682 616905 60 61
tig00000001_5 62620 28879 60 61
tig00000010 59063 683695 60 61
tig00000047 50189 1228806 60 61
tig00000023 50158 908211 60 61
tig00000413 46361 2760317 60 61
tig00000031 45324 1069679 60 61
tig00000003_5 45045 192121 60 61
tig00000029 43222 1025723 60 61
Running the following, and then restarting the snakemake pipeline allowed the pipeline to complete.
mkdir -p Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/
touch Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/dummy
Hi @brymerr921
Did you get the solution to your problem? Cuz I am facing a similar issue where it says :
Activating singularity image /lustre/rsharma/.snakemake/singularity/f075a3ce7bc1893be5302161e5e7807a.simg
Waiting at most 30 seconds for missing files.
MissingOutputException in line 574 of /home/rsharma/lathe/Snakefile:
Job Missing files after 30 seconds:
W1/3.circularization/1.candidate_genomes
This might be due to filesystem latency. If that is the case, consider increasing the wait time with --latency-wait.
Job id: 25 completed successfully, but some output files are missing. 25
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /lustre/rsharma/.snakemake/log/2021-08-12T150046.064606.snakemake.log
{'W1': ['/home/rsharma/WWK_forward_paired.fastq', '/home/rsharma/WWK_reverse_paired.fastq']}
Checking for updates or modifications to workflow
WARNING: Differences to latest version detected. Please reset changes and/or pull repo.
Saraswati
@brymerr921
Do I have to run the whole pipeline again or I can start from circularization?