85ya / fastDNA

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Table of contents

Introduction

fastDNA is a library for classification of short DNA sequences. It is adapted from the fastText library.

Requirements

Generally, fastText builds on modern Mac OS and Linux distributions. Since it uses some C++11 features, it requires a compiler with good C++11 support. These include :

  • (g++-4.7.2 or newer) or (clang-3.3 or newer)

Compilation is carried out using a Makefile, so you will need to have a working make.

Building fastDNA

$ git clone https://github.com/rmenegaux/fastDNA.git
$ cd fastDNA
$ make

This will produce object files for all the classes as well as the main binary fastdna.

For a trial run:

$ cd test
$ sh test.sh

This should train and evaluate a small model on the toy dataset provided.

DNA short read classification

In order to train a dna classifier using the method described in 1, use:

$ ./fastdna supervised -input train.fasta -labels labels.txt -output model

where train.fasta is a FASTA file containing the full reference genomes and labels.txt is a text file containing the genome labels (one label per line). This will output two files: model.bin and model.vec.

Once the model was trained, you can evaluate it by computing the precision and recall at k (P@k and R@k) on a test set using:

$ ./fastdna test model.bin test.fasta test_labels.txt n

where test.fasta is a FASTA file containing the DNA fragments to be classified, and test_labels.txt contains the labels for each of the fragments.

The argument n is optional, and is equal to 1 by default.

In order to obtain the n most likely labels for a set of reads, use:

$ ./fastdna predict model.bin test.fasta n

or use predict-prob to also get the probability for each label

$ ./fastdna predict-prob model.bin test.fasta n

Doing so will print to the standard output the n most likely labels for each line. The argument n is optional, and equal to 1 by default.

If you want to compute vector representations of DNA sequences, please use:

$ ./fastdna print-word-vectors model.bin < text.fasta

This assumes that the text.fasta file contains the DNA sequences that you want to get vectors for. The program will output one vector representation per sequence in the file.

To write the vectors to a file, redirect the output as so:

$ ./fastdna print-word-vectors model.bin < text.fasta > vectors.txt

To get vectors from standard input, just type

$ ./fastdna print-word-vectors model.bin

Press Enter, then type the sequence and finish with Ctrl+D (Linux, Mac) or Ctrl+Z (Windows)

You can also quantize a supervised model to reduce its memory usage with the following command:

$ ./fastdna quantize -output model

This will create a .ftz file with a smaller memory footprint. All the standard functionality, like test or predict work the same way on the quantized models:

$ ./fastdna test model.ftz test.fasta test_labels.txt

The quantization procedure follows the steps described in 3.

Full documentation

Invoke a command without arguments to list available arguments and their default values:

$ ./fastdna supervised
Empty input or output path.

The following arguments are mandatory:
  -input              training file path
  -output             output file path

The following arguments are optional:
  -verbose            verbosity level [2]

The following arguments for the dictionary are optional:
  -minn               min length of char ngram [0]
  -maxn               max length of char ngram [0]
  -label              labels prefix [__label__]

The following arguments for training are optional:
  -lr                 learning rate [0.1]
  -lrUpdateRate       change the rate of updates for the learning rate [100]
  -dim                size of word vectors [100]
  -noise              mutation rate (/100,000)[0]
  -length             length of fragments for training [200]
  -epoch              number of epochs [5]
  -loss               loss function {ns, hs, softmax} [softmax]
  -thread             number of threads [12]
  -pretrainedVectors  pretrained word vectors for supervised learning []
  -loadModel          pretrained model for supervised learning []
  -saveOutput         whether output params should be saved [false]
  -freezeEmbeddings   model does not update the embedding vectors [false]

The following arguments for quantization are optional:
  -cutoff             number of words and ngrams to retain [0]
  -retrain            whether embeddings are finetuned if a cutoff is applied [false]
  -qnorm              whether the norm is quantized separately [false]
  -qout               whether the classifier is quantized [false]
  -dsub               size of each sub-vector [2]

Python

Most use cases are covered in the python script fdna.py.

To reproduce the results from the paper, download the data then run:

python fdna.py -train -train_fasta /path/to/train_large_fasta -train_labels /path/to/train_large_labels \
    -eval -test_fasta /path/to/test_large_fasta -test_labels /path/to/test_large_labels \
    -k 13 -d 100 -noise 4 -e 200

NB: Best parameters for classification tasks are k=14, d=50, noise=4

Full usage:

python fdna.py --help
usage: fdna.py [-h] [-train] [-quantize] [-predict] [-eval] [-predict_quant]
               [-train_fasta TRAIN_FASTA] [-train_labels TRAIN_LABELS]
               [-test_fasta TEST_FASTA] [-test_labels TEST_LABELS]
               [-output_dir OUTPUT_DIR] [-model_name MODEL_NAME]
               [-threads THREADS] [-d D] [-k K] [-e E] [-lr LR] [-noise NOISE]
               [-L L] [-freeze] [-pretrained_vectors PRETRAINED_VECTORS]
               [-verbose VERBOSE]

train, predict and/or quantize fdna model

optional arguments:
  -h, --help            show this help message and exit
  -train                train model
  -quantize             quantize model
  -predict              make predictions
  -eval                 make and evaluate predictions
  -predict_quant        make and evaluate predictions with quantized model
  -train_fasta TRAIN_FASTA
                        training dataset, fasta file containing full genomes
  -train_labels TRAIN_LABELS
                        training labels, text file containing as many labels
                        as there are training genomes
  -test_fasta TEST_FASTA
                        testing dataset, fasta file containing reads
  -test_labels TEST_LABELS
                        testing dataset, text file containing as many labels
                        as there are reads
  -output_dir OUTPUT_DIR
                        output directory
  -model_name MODEL_NAME
                        optional user-defined model name
  -threads THREADS      number of threads
  -d D                  embedding dimension
  -k K                  k-mer length
  -e E                  number of training epochs
  -lr LR                learning rate
  -noise NOISE          level of training noise, percent of random mutations
  -L L                  training read length
  -freeze               freeze the embeddings
  -pretrained_vectors PRETRAINED_VECTORS
                        pretrained vectors .vec files
  -verbose VERBOSE      output verbosity, 0 1 or 2

The python scripts require numpy and scikit-learn for evaluating predictions.

Data

The data used in the paper is available here: http://projects.cbio.mines-paristech.fr/largescalemetagenomics/.

The small and large datasets used in the paper can be found here

References

Continuous Embedding of DNA reads, and application to metagenomics

[1] R. Menegaux, J. Vert, Continuous Embedding of DNA reads, and application to metagenomics

@article{menegaux2018continuous,
  title={Continuous Embedding of DNA reads and application to metagenomics},
  author={Menegaux, Romain and Vert, Jean-Philippe},
  journal={bioRxiv preprint 335943},
  year={2018}
}

Enriching Word Vectors with Subword Information

[2] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching Word Vectors with Subword Information

@article{bojanowski2016enriching,
  title={Enriching Word Vectors with Subword Information},
  author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.04606},
  year={2016}
}

Bag of Tricks for Efficient Text Classification

[3] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of Tricks for Efficient Text Classification

@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}

FastText.zip: Compressing text classification models

[4] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, T. Mikolov, FastText.zip: Compressing text classification models

@article{joulin2016fasttext,
  title={FastText.zip: Compressing text classification models},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J{\'e}gou, H{\'e}rve and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1612.03651},
  year={2016}
}

License

fastText is BSD-licensed. An additional patent grant is also provided

About

License:Other


Languages

Language:C++ 87.9%Language:Python 10.3%Language:Makefile 1.2%Language:Shell 0.7%