Repository for the deep learning models I used in my 2019 summer vacation research at UNSW
e18MouseData.py
provides a Dataset class E18MouseData
which can be used create a PyTorch friendly Dataset from GSE93421_bbrain_aggregate_matrix.hdf5.
This code is intended to be used with GSE93421_brain_aggregate_matrix.hdf5 (ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE93nnn/GSE93421/suppl/GSE93421_brain_aggregate_matrix.hdf5). Further information is available here; however, I have been unable to find a thorough description detailing how this dataset is organized.
The following sections represent my best guess at the dataset's structure. The hdf5 file contains 7 1D lists under the head node 'mm10'.
Barcode identifier for each sequenced cell
Count data. Each entry corresponds to a reading for a specific gene and cell. See below for details ...
Gene identifiers.
This list has a 1-1 correspondence with data. Each entry represents an index in genes
/gene_names
(0 <= v < 27998). It indicates what gene the corresponding entry in data
is refering to.
This is has a 1-1 correspondence with barcodes
. Each entry ris a pointer to an index in data
(monotonically increasing with 0 <= v <~ 2.6 billion). Each entry in data
between two consective values of indptr
are count data for the same cell with the corresponding gene given by indices
.
This dataset is very large, especially in it's full sparse representation (~36 billion datapoints). This code will require approximately 170GB of RAM to load the full dataset (I provide the option to only load a fraction of it in). It takes about 15 minutes to load even using 20 processes in parallel on a dual socket Intel E5-2699 (2.2GHz).
- My fork of pt-sdae (including a branch compatible with PyTorch 0.35 for Cuda 7.5)
- h5py (for loading in data)
- sharedmem (to allow large shared-memory numpy arrays between processes)
- sklearn (for tsne)
- MulticoreTSNE (for multicore compatible tsne)
- umap-learn (for umap)