convert vcf to plink step missing id setting

Question

convert vcf to plink step missing id setting

trptyrphe11 opened this issue 6 years ago · comments

Hi,

I found using the current command will result in all variant id specified as "." in the bim file. Is that wrong? Shall we set variant id manually using the parameter --set-missing-var-ids?

Andries Marees · Answer 1 · Fri Sep 21 2018 17:19:47 GMT+0800 (China Standard Time)

Dear Sir/Madam, Thank you for your email. My apologies for replying so late to it (I did not see it earlier, since your email went into my spam folder). It is not wrong that some (not all) variant ids are specified as ".". If you look into the file: ALL.2of4intersection.20100804.genotypes.vcf.gz, by using, for example: zmore ALL.2of4intersection.20100804.genotypes.vcf.gz [use space to scroll through the file] you see that the unmodified vcf file (from 1000 Genomes) also contains many ".". So, the bim file you mention only contains a variant ID (i.e., rs-identifier) where this is specified in the 1000Genomes data. The variants without an rs-identifier are indicated with ".", as you observed. For this tutorial however, the 'missing' rs-identifiers are not important. Please let me know if you have any further questions. Best regards, Andries

…

________________________________ Van: trptyrphe11 <notifications@github.com> Verzonden: vrijdag 7 september 2018 21:14 Aan: MareesAT/GWA_tutorial CC: Subscribed Onderwerp: [MareesAT/GWA_tutorial] convert vcf to plink step missing id setting (#1) Hi, I found using the current command will result in all variant id specified as "." in the bim file. Is that wrong? Shall we set variant id manually using the parameter --set-missing-var-ids? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#1>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ANKHmgDJdbfRh5c3mf3uj7SZfWy4yGjBks5uYsWqgaJpZM4WfVwn>.

trptyrphe11 · Answer 2 · Fri Sep 21 2018 20:31:08 GMT+0800 (China Standard Time)

Hi Andries , So if multiple variants have the same identifier '.', how downstream analysis (such as LD pruning, when the prune.in file has only . in the rows) distinguish them? I have tried with or without --set-missing-var-ids and the results are different. Thanks. Best, Huilei

…

On Fri, Sep 21, 2018 at 5:19 AM MareesAT ***@***.***> wrote: Dear Sir/Madam, Thank you for your email. My apologies for replying so late to it (I did not see it earlier, since your email went into my spam folder). It is not wrong that some (not all) variant ids are specified as ".". If you look into the file: ALL.2of4intersection.20100804.genotypes.vcf.gz, by using, for example: zmore ALL.2of4intersection.20100804.genotypes.vcf.gz [use space to scroll through the file] you see that the unmodified vcf file (from 1000 Genomes) also contains many ".". So, the bim file you mention only contains a variant ID (i.e., rs-identifier) where this is specified in the 1000Genomes data. The variants without an rs-identifier are indicated with ".", as you observed. For this tutorial however, the 'missing' rs-identifiers are not important. Please let me know if you have any further questions. Best regards, Andries ________________________________ Van: trptyrphe11 ***@***.***> Verzonden: vrijdag 7 september 2018 21:14 Aan: MareesAT/GWA_tutorial CC: Subscribed Onderwerp: [MareesAT/GWA_tutorial] convert vcf to plink step missing id setting (#1) Hi, I found using the current command will result in all variant id specified as "." in the bim file. Is that wrong? Shall we set variant id manually using the parameter --set-missing-var-ids? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< #1>, or mute the thread< https://github.com/notifications/unsubscribe-auth/ANKHmgDJdbfRh5c3mf3uj7SZfWy4yGjBks5uYsWqgaJpZM4WfVwn >. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMEG0NHUk7rT742FyzsGaRn5kWdjPYaZks5udK81gaJpZM4WfVwn> .

Andries Marees · Answer 3 · Sat Sep 22 2018 03:05:13 GMT+0800 (China Standard Time)

Hi Huilei, Thank you for your reply and making me aware of this. I do agree that it is good practice to include the command --set-missing-var-ids. Therefore, I included this command in the most recent version of in the tutorial. (The inclusion of this command does not change the results when exactly following the script in this tutorial. After the QC of the 1000 Genomes dataset we generated a file with only SNPs present in both the HapMap and 1000 Genomes datasets, the content of this file does not change by using the --set-missing-var-ids command. In this merged dataset all SNPs have a rs-identifier. The pruning step is conducted on this merged dataset, which does not include any variants with "." as identifier, as all variants have an rs-identifier.) If you have any other remarks or questions please let me know. Best regards, Andries

________________________________ Van: trptyrphe11 <notifications@github.com> Verzonden: vrijdag 21 september 2018 14:31 Aan: MareesAT/GWA_tutorial CC: MareesAT; Comment Onderwerp: Re: [MareesAT/GWA_tutorial] convert vcf to plink step missing id setting (#1) Hi Andries , So if multiple variants have the same identifier '.', how downstream analysis (such as LD pruning, when the prune.in file has only . in the rows) distinguish them? I have tried with or without --set-missing-var-ids and the results are different. Thanks. Best, Huilei

On Fri, Sep 21, 2018 at 5:19 AM MareesAT ***@***.***> wrote: Dear Sir/Madam, Thank you for your email. My apologies for replying so late to it (I did not see it earlier, since your email went into my spam folder). It is not wrong that some (not all) variant ids are specified as ".". If you look into the file: ALL.2of4intersection.20100804.genotypes.vcf.gz, by using, for example: zmore ALL.2of4intersection.20100804.genotypes.vcf.gz [use space to scroll through the file] you see that the unmodified vcf file (from 1000 Genomes) also contains many ".". So, the bim file you mention only contains a variant ID (i.e., rs-identifier) where this is specified in the 1000Genomes data. The variants without an rs-identifier are indicated with ".", as you observed. For this tutorial however, the 'missing' rs-identifiers are not important. Please let me know if you have any further questions. Best regards, Andries ________________________________ Van: trptyrphe11 ***@***.***> Verzonden: vrijdag 7 september 2018 21:14 Aan: MareesAT/GWA_tutorial CC: Subscribed Onderwerp: [MareesAT/GWA_tutorial] convert vcf to plink step missing id setting (#1) Hi, I found using the current command will result in all variant id specified as "." in the bim file. Is that wrong? Shall we set variant id manually using the parameter --set-missing-var-ids? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< #1>, or mute the thread< https://github.com/notifications/unsubscribe-auth/ANKHmgDJdbfRh5c3mf3uj7SZfWy4yGjBks5uYsWqgaJpZM4WfVwn >. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMEG0NHUk7rT742FyzsGaRn5kWdjPYaZks5udK81gaJpZM4WfVwn> .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ANKHml0Yn4p3kw40A5R4ng5oy4dE7glJks5udNwNgaJpZM4WfVwn>.

ttbek · Answer 4 · Tue Aug 06 2019 23:33:25 GMT+0800 (China Standard Time)

In general, the best way to convert from VCF to plink is to split multi-allelic sites, left align/normalize, give unique IDs, and then convert. This is described here: http://apol1.blogspot.com/2014/11/best-practice-for-converting-vcf-files.html

The command is:

  bcftools norm -Ou -m -any input.vcf.gz |
  bcftools norm -Ou -f human_g1k_v37.fasta |
  bcftools annotate -Ob -x ID \
    -I +'%CHROM:%POS:%REF:%ALT' |
  plink --bcf /dev/stdin \
    --keep-allele-order \
    --vcf-idspace-to _ \
    --const-fid \
    --allow-extra-chr 0 \
    --split-x b37 no-fail \
    --make-bed \
    --out output

This requires bcftools 1.9 and Plink 1.9 or 2.0 (is still alpha as of Aug 6th, 2019), it also requires you to have the reference genome in fasta format (.fa or .fasta).
Setting the IDs this way also lets you keep track of which is ref vs. alt.

rs numbers as IDs are kind of wonky in general despite being a very common practice, please see the discussion freeseek references in his/her blog post (http://annovar.openbioinformatics.org/en/latest/articles/dbSNP/) written by the author of Annovar.

Normalization is well described here: https://genome.sph.umich.edu/wiki/Variant_Normalization

It's pretty cool that bcftools can be piped that way, but sometimes you want to keep the intermediate files, the tee utility will let you save these, it writes what it gets on standard in to a file while also to standard out for the next pipe, e.g.

echo "Testing, 1, 2, 3..." | tee ./test_file.txt | grep Test

Will both show the output of grep on screen and write "Testing, 1, 2, 3..." to the file test_file.txt.