segmentation fault using ngmlr 0.2.7

Question

segmentation fault using ngmlr 0.2.7

lentil824 opened this issue 3 years ago · comments

Hi there,

I installed precompiled ngmlr 0.2.7, copied the demo command for Nanopore run, and replaced with my own ref.fa and whole genome seq.fastq.gz files. It immediately gave "segmentation fault" error.

I don't have permission type issue (#26), and ngmlr version not the old one (#10). I even zcat the gz file, copied the ref.fa to the current dir, none was working. Always the same error. Did I miss something?

Thanks,
Susan

Chris Winefield · Answer 1 · Thu Jun 17 2021 08:02:46 GMT+0800 (China Standard Time)

Hi Susan,
I have also just installed this version of ngmlr
and am experiencing the same issues - I was just wondering if you have made any progress or have moved to another approach in the meantime?

Regards
Chris

lentil824 · Answer 2 · Thu Jun 24 2021 01:45:21 GMT+0800 (China Standard Time)

Hi Chris,

Sorry for the late reply! Unfortunately I couldn't figure this out and didn't get a reply from the developer, so I switched to other mappers (https://github.com/lh3/minimap2 and https://github.com/marbl/Winnowmap). Both minimap2 and winnowmap are easy to install and use, so you can give them a try.

Regards,
Susan

Chris Winefield · Answer 3 · Thu Jun 24 2021 05:01:48 GMT+0800 (China Standard Time)

Thanks Susan, I'm in the same boat and have gone to minimap2. Other issues have arisen with my collaborators with them providing the PacBio reads in a format that is now causing other issues - I really love bioinformatics at times 😐. Thanks for the note - Much appreciated Kind Regards Chris Chris Winefield Associate Professor in Plant Genomics and Molecular Biology Department of Wine Food and Molecular Biosciences Faculty of Agriculture and Life Sciences RFH, room 062 Engineering Drive P O Box 85084 Lincoln University Lincoln 7647 Christchurch New Zealand p +64 3 4230630 | m +64 021 0238 4476 e ***@***.*** | w www.lincoln.ac.nz<http://www.lincoln.ac.nz/> Lincoln University, Te Whare Wānaka o Aoraki New Zealand's Specialist Land-Based University [cid:167c3081-d879-4de5-8427-b237888bf720] The NZ Plant <http://www.mobilekiwi.org/> Transposon <http://www.mobilekiwi.org/> Team<http://www.mobilekiwi.org/>

…

________________________________ From: lentil824 ***@***.***> Sent: 24 June 2021 5:45 AM To: philres/ngmlr ***@***.***> Cc: Winefield, Christopher ***@***.***>; Comment ***@***.***> Subject: Re: [philres/ngmlr] segmentation fault using ngmlr 0.2.7 (#94) Hi Chris, Sorry for the late reply! Unfortunately I couldn't figure this out and didn't get a reply from the developer, so I switched to other mappers (https://github.com/lh3/minimap2<https://github.com/lh3/minimap2> and https://github.com/marbl/Winnowmap<https://github.com/marbl/Winnowmap>). Both minimap2 and winnowmap are easy to install and use, so you can give them a try. Regards, Susan — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#94 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABYMLSSR2L7YIIZPKWSHAK3TUIMTXANCNFSM4ZD6FAUA>.

________________________________ "The contents of this e-mail (including any attachments) may be confidential and/or subject to copyright. Any unauthorised use, distribution, or copying of the contents is expressly prohibited. If you have received this e-mail in error, please advise the sender by return e-mail or telephone and then delete this e-mail together with all attachments from your system."

Fritz Sedlazeck · Answer 4 · Thu Jun 24 2021 05:04:16 GMT+0800 (China Standard Time)

Sorry guys for the super late reply. Can you describe a bit more about the error that you see and the output that you get from NGMLR running on the command line ?
Thanks
Fritz

Chris Winefield · Answer 5 · Thu Jun 24 2021 05:52:26 GMT+0800 (China Standard Time)

Hi Fritz, Not to worry. I have been working with our national HPC cluster. I have installed ngmlr as the precompiled version into my project space and point the SLUM scheduler to this from my scratch/nobackup space (where the data sits as well). The SLURM script to launch the analysis is: #!/bin/bash -e #SBATCH --job-name=Nbenth_ngmlr_align #SBATCH --output=AW_%j.out #SBATCH --error=AW_%j.err #SBATCH ***@***.*** #SBATCH --mail-type=END #SBATCH --time=02:00:00 #SBATCH --mem=80G #SBATCH --partition=bigmem #SBATCH --ntasks=10 #SBATCH --profile=task /nesi/project/lincoln03032/ngmlr/ngmlr/bin/ngmlr-0.2.8/ngmlr -t 20 -r\ /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/NbQld082.genome.fasta -q\ /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/LABallPacBio.fasta -o\ /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/ngmlr_QLD_aligned_with_LAB.sam the .err file outputs the following after about 12 sec to 3 minutes depending on the run ***@***.*** data]$ cat AW_20467022.err ngmlr 0.2.8 (build: Jun 16 2021 22:04:17, start: 2021-06-16.22:27:39) Contact: ***@***.***, ***@***.*** Opening for output (SAM): QLD_aligned_with_LAB.sam Encoding reference sequence. Size of reference genome 2912 Mbp (max. 68719 Mbp) 0 reference sequences were skipped (length < 10). Writing encoded reference to NbQld082.genome.fasta-enc.2.ngm Writing to disk took 1.34s Building reference index #0 (kmer length: 13, reference skip: 2) 81228 prefixes were ignored due to the frequency cutoff (1000) Overall time for creating RefTable: 386.50s Writing reference index to NbQld082.genome.fasta-ht-13-2.2.ngm Writing to disk took 5.83s Opening query file LABallPacBio.fasta Mapping reads... /var/spool/slurm/job20467022/slurm_script: line 13: 37953 Segmentation fault (core dumped) /nesi/project/lincoln03032/ngmlr/ngmlr/bin/ngmlr-0.2.8/ngmlr -t 4 -r NbQld082.genome.fasta -q LABallPacBio.fasta -o QLD_aligned_with_LAB.sam So the package loads and appears to start OK then gives a segmentation fault and core dump as the mapping begins. a couple of thoughts - Should I have installed the ngmlr programme in the same folder as I have the data and where the output will end up? The other thought was I hadn't set the number of threads in SLUM as the same as the -t call for ngmlr - I tried this a day or so ago and this didnt appear to help. However, I have since found that the PacBio data I was given isnt complete. The fasta file I have to work with is the striped version of the raw PacBio BAM file so only contains read ID and data. One of the files also mixes ONT data into the mix - which was frustrating to find out! I am wondering if this might be causing an error as I expect ngmlr will be looking for read quality metrics associated with the seq data type (PacBio/ONT/illumina). As an aside would ngmlr accept the raw PacBio BAM file format? Thanks for the help with tis Regards Chris Chris Winefield Associate Professor in Plant Genomics and Molecular Biology Department of Wine Food and Molecular Biosciences Faculty of Agriculture and Life Sciences RFH, room 062 Engineering Drive P O Box 85084 Lincoln University Lincoln 7647 Christchurch New Zealand p +64 3 4230630 | m +64 021 0238 4476 e ***@***.*** | w www.lincoln.ac.nz<http://www.lincoln.ac.nz/> Lincoln University, Te Whare Wānaka o Aoraki New Zealand's Specialist Land-Based University [cid:4180d74c-65a4-4c64-b17c-6a12f83a1ace] The NZ Plant <http://www.mobilekiwi.org/> Transposon <http://www.mobilekiwi.org/> Team<http://www.mobilekiwi.org/>

…

________________________________ From: Fritz Sedlazeck ***@***.***> Sent: 24 June 2021 9:04 AM To: philres/ngmlr ***@***.***> Cc: Winefield, Christopher ***@***.***>; Comment ***@***.***> Subject: Re: [philres/ngmlr] segmentation fault using ngmlr 0.2.7 (#94) Sorry guys for the super late reply. Can you describe a bit more about the error that you see and the output that you get from NGMLR running on the command line ? Thanks Fritz — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#94 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABYMLSQVRSLMJB3YXAIHLP3TUJD5VANCNFSM4ZD6FAUA>.

________________________________ "The contents of this e-mail (including any attachments) may be confidential and/or subject to copyright. Any unauthorised use, distribution, or copying of the contents is expressly prohibited. If you have received this e-mail in error, please advise the sender by return e-mail or telephone and then delete this e-mail together with all attachments from your system."

Fritz Sedlazeck · Answer 6 · Thu Jun 24 2021 05:55:24 GMT+0800 (China Standard Time)

That should not happen like that..
I would suggest that maybe if you can take the test data from this repo and run it with that once . Like e.g. test1 should show you if you get it there or not.

Thx
Fritz

Chris Winefield · Answer 7 · Thu Jun 24 2021 05:57:04 GMT+0800 (China Standard Time)

Hi Fritz, I will grab the data from the repo and give this a whirl and see where we land. Back to you shortly Regards Chris Chris Winefield Associate Professor in Plant Genomics and Molecular Biology Department of Wine Food and Molecular Biosciences Faculty of Agriculture and Life Sciences RFH, room 062 Engineering Drive P O Box 85084 Lincoln University Lincoln 7647 Christchurch New Zealand p +64 3 4230630 | m +64 021 0238 4476 e ***@***.*** | w www.lincoln.ac.nz<http://www.lincoln.ac.nz/> Lincoln University, Te Whare Wānaka o Aoraki New Zealand's Specialist Land-Based University [cid:3290ca68-f878-43db-ae7a-9813f5f0e9a6] The NZ Plant <http://www.mobilekiwi.org/> Transposon <http://www.mobilekiwi.org/> Team<http://www.mobilekiwi.org/>

…

________________________________ From: Fritz Sedlazeck ***@***.***> Sent: 24 June 2021 9:55 AM To: philres/ngmlr ***@***.***> Cc: Winefield, Christopher ***@***.***>; Comment ***@***.***> Subject: Re: [philres/ngmlr] segmentation fault using ngmlr 0.2.7 (#94) That should not happen like that.. I would suggest that maybe if you can take the test data from this repo and run it with that once . Like e.g. test1 should show you if you get it there or not. Thx Fritz — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#94 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABYMLSRNES7N2T6NDSXVVJTTUJJ5NANCNFSM4ZD6FAUA>.

________________________________ "The contents of this e-mail (including any attachments) may be confidential and/or subject to copyright. Any unauthorised use, distribution, or copying of the contents is expressly prohibited. If you have received this e-mail in error, please advise the sender by return e-mail or telephone and then delete this e-mail together with all attachments from your system."

Chris Winefield · Answer 8 · Thu Jun 24 2021 06:33:00 GMT+0800 (China Standard Time)

Hi Fritz, With the test_1 data it appears to work fine. ngmlr 0.2.8 (build: Jun 16 2021 22:04:17, start: 2021-06-23.22:19:56) Contact: ***@***.***, ***@***.*** Opening for output (SAM): /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/test1_ngmlr.sam Encoding reference sequence. Size of reference genome 0 Mbp (max. 68719 Mbp) 0 reference sequences were skipped (length < 10). Writing encoded reference to /nesi/project/lincoln03032/ngmlr/ngmlr/test/data/test_1/ref_chr6_140kb.fa-enc.2.ngm Writing to disk took 0.10s Building reference index #0 (kmer length: 13, reference skip: 2) 0 prefixes were ignored due to the frequency cutoff (1000) Overall time for creating RefTable: 2.88s Writing reference index to /nesi/project/lincoln03032/ngmlr/ngmlr/test/data/test_1/ref_chr6_140kb.fa-ht-13-2.2.ngm Writing to disk took 0.21s Opening query file /nesi/project/lincoln03032/ngmlr/ngmlr/test/data/test_1/long_name.fa Mapping reads... Processed: 3 (1.00), R/S: 1.50, RL: 3933, Time: 3.00 7.00 34.50, Align: 1.00, 309, 1.00 Done (3 reads mapped (100.00%), 0 reads not mapped, 3 lines written)(elapsed: 0m, 0 r/s) So I am wondering if the fault I am having is a result of the format of the read/seq files? I had a look at your test data and the data I have been trying to map and there is quite a difference (no suprises there as the data I have been working with is essentially an extracted fasta from the originating BAM) My data:

ba0503ce-f9cb-41aa-8a75-ff8942291e67 runid=e8bded7f079909caf709756860f919e80edb7f0d sampleid=L1 read=127 ch=2174 start_time=2018-07-02T22:49:03Z

ATGATGTGCGCTTCGTTCAGTTACACCATCAGATTGTGTTAGTCTTTTTTTTTGGAATTTTTGAATTTTTGCCAACCTCTGCCGTTTGCCGTGCATATCGGTCACGAACAGTCAATTGCAAACTGGTAACCTGGATTTGTTCTATCAGTAATCGACCTTGTCCCTAATTAAATCGAATAAATCCTTA your test data:

m140612_020550_42156_c100652082550000001823118110071460_s1_p0/49/0_11124>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/66/1019_13505>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/66/13545_26663>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/66/26699_39808>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/157671/0_4333>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/157674/0_6303>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/157676/0_14740>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/157677/3585_9506>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/157664/5106_6916>m140612_020550_42156_c100652082550000001823118110071460_s1_p0/157668/1089_6689

AAAATATCAGTAAAATCAGATTGACCTCAAACCCTGTATCTTTCAAAAAAGACATAATTTTGTTTTTCAAGCA I will get the chaps in Australia to provide the raw files and I will try again. Regards Chris Chris Winefield Associate Professor in Plant Genomics and Molecular Biology Department of Wine Food and Molecular Biosciences Faculty of Agriculture and Life Sciences RFH, room 062 Engineering Drive P O Box 85084 Lincoln University Lincoln 7647 Christchurch New Zealand p +64 3 4230630 | m +64 021 0238 4476 e ***@***.*** | w www.lincoln.ac.nz<http://www.lincoln.ac.nz/> Lincoln University, Te Whare Wānaka o Aoraki New Zealand's Specialist Land-Based University [cid:eadb1b58-07a6-4ffc-9484-1119a3d1ba83] The NZ Plant <http://www.mobilekiwi.org/> Transposon <http://www.mobilekiwi.org/> Team<http://www.mobilekiwi.org/>

…

________________________________ From: Fritz Sedlazeck ***@***.***> Sent: 24 June 2021 9:55 AM To: philres/ngmlr ***@***.***> Cc: Winefield, Christopher ***@***.***>; Comment ***@***.***> Subject: Re: [philres/ngmlr] segmentation fault using ngmlr 0.2.7 (#94) That should not happen like that.. I would suggest that maybe if you can take the test data from this repo and run it with that once . Like e.g. test1 should show you if you get it there or not. Thx Fritz — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#94 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABYMLSRNES7N2T6NDSXVVJTTUJJ5NANCNFSM4ZD6FAUA>.

________________________________ "The contents of this e-mail (including any attachments) may be confidential and/or subject to copyright. Any unauthorised use, distribution, or copying of the contents is expressly prohibited. If you have received this e-mail in error, please advise the sender by return e-mail or telephone and then delete this e-mail together with all attachments from your system."

Fritz Sedlazeck · Answer 9 · Thu Jun 24 2021 06:34:25 GMT+0800 (China Standard Time)

Thanks . Feel free to keep me posted.
Fritz

lentil824 · Answer 10 · Thu Jun 24 2021 08:06:31 GMT+0800 (China Standard Time)

Hi Fritz,

Thanks for the suggestion! The same error "segmentation fault" appeared when using test_1 data and precompiled ngmlr 0.2.7. I noticed Chris was using ngmlr 0.2.8 build, so decided to build ngmlr from source. Surprisingly, ngmlr 0.2.8 build from source worked well with test_1 data and my own data (still running so far with my large data file). So, now I think there might be some issues with the precompiled version, but I will leave this to you.

Regards,
Susan

lentil824 · Answer 11 · Thu Jun 24 2021 08:20:36 GMT+0800 (China Standard Time)

Btw, Chris, I have ONT data in fasta. So it is pretty straightforward by adding -x ont. Your data is a bit complicated from what you described. With the help from Fritz, I am sure you will get it sorted out.

Regards,
Susan

Fritz Sedlazeck · Answer 12 · Tue Jun 29 2021 04:18:04 GMT+0800 (China Standard Time)

Thanks Susan!
Fritz

Chris Winefield · Answer 13 · Wed Jul 07 2021 12:08:33 GMT+0800 (China Standard Time)

Hi Fritz, Sorry for the delay in getting back to you - it has taken a while to get the raw data files from my collaborators. I have tried a few different things, including moving the instal site to the same partition as the raw data. I have just tried mapping the fasta packaged with the PacBio tarball I received. The run fails again with a segmentation fault but this time generated some mapped reads before reporting a invalid pointer and mem corruption the output from the .err file is as follows: ngmlr 0.2.8 (build: Jul 1 2021 04:45:58, start: 2021-07-07.03:52:21) Contact: ***@***.***, ***@***.*** Opening for output (SAM): /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/ngmlr_LAB_aligned_with_QLD_data.sam Encoding reference sequence. Size of reference genome 2836 Mbp (max. 68719 Mbp) 0 reference sequences were skipped (length < 10). Writing encoded reference to /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/NbLab330.genome.fasta-enc.2.ngm Writing to disk took 0.46s Building reference index #0 (kmer length: 13, reference skip: 2) 82438 prefixes were ignored due to the frequency cutoff (1000) Overall time for creating RefTable: 324.03s Writing reference index to /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/NbLab330.genome.fasta-ht-13-2.2.ngm Writing to disk took 1.19s Opening query file /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/QLD_Benth_1/m54105_190325_053810.fasta Mapping reads... Processed: 7 (0.57), R/S: 3.50, RL: 681342, Time: 0.00 0.00 0.00, Align: 1.00, 1211, 0.90 *** Error in `ngmlr': free(): invalid pointer: 0x00002aad2c0bb6f0 *** *** Error in `ngmlr': malloc(): memory corruption: 0x00002aad2c0bb730 *** /var/spool/slurm/job20732151/slurm_script: line 13: 15624 Segmentation fault (core dumped) /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/ngmlr/bin/ngmlr-0.2.8/ngmlr -t 20 -r /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/NbLab330.genome.fasta -q /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/QLD_Benth_1/m54105_190325_053810.fasta -o /nesi/nobackup/lincoln03032/TE_analysis/nbenth_analysis/data/ngmlr_LAB_aligned_with_QLD_data.sam Any thoughts of where to go from here would be fantastic. Kind Regards Chris Chris Winefield Associate Professor in Plant Genomics and Molecular Biology Department of Wine Food and Molecular Biosciences Faculty of Agriculture and Life Sciences RFH, room 062 Engineering Drive P O Box 85084 Lincoln University Lincoln 7647 Christchurch New Zealand p +64 3 4230630 | m +64 021 0238 4476 e ***@***.*** | w www.lincoln.ac.nz<http://www.lincoln.ac.nz/> Lincoln University, Te Whare Wānaka o Aoraki New Zealand's Specialist Land-Based University [cid:5cee39f5-5cf1-4045-b73b-bf66bd7a24f3] The NZ Plant <http://www.mobilekiwi.org/> Transposon <http://www.mobilekiwi.org/> Team<http://www.mobilekiwi.org/>

…

________________________________ From: Fritz Sedlazeck ***@***.***> Sent: 24 June 2021 10:34 AM To: philres/ngmlr ***@***.***> Cc: Winefield, Christopher ***@***.***>; Comment ***@***.***> Subject: Re: [philres/ngmlr] segmentation fault using ngmlr 0.2.7 (#94) Thanks . Feel free to keep me posted. Fritz — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#94 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABYMLSWESDWVIJ3YXRI3A5LTUJOPXANCNFSM4ZD6FAUA>.

________________________________ "The contents of this e-mail (including any attachments) may be confidential and/or subject to copyright. Any unauthorised use, distribution, or copying of the contents is expressly prohibited. If you have received this e-mail in error, please advise the sender by return e-mail or telephone and then delete this e-mail together with all attachments from your system."