Ensemble Integration (EI): Integrating multimodal data through interpretable heterogeneous ensembles
The latest version of EI fully written in python is implemented here, or you may install it by pip install ensemble-integration with full documentation.
Ensemble Integration (EI) is a customizable pipeline for generating diverse ensembles of heterogeneous classifiers, as well as the accompanying metadata needed for ensemble learning approaches utilizing ensemble diversity for improved performance. It also fairly evaluates the performance of several ensemble learning methods including ensemble selection [Caruana2004], and stacked generalization (stacking) [Wolpert1992]. Though other tools exist, we are unaware of a similarly modular, scalable pipeline designed for large-scale ensemble learning. EI was developed to support research by Yan Chak Li, Linhua Wang, and Gaurav Pandey.
EI is designed for generating extremely large ensembles (taking days or weeks to generate) and thus consists of an initial data generation phase tuned for multicore and distributed computing environments. The output is a set of compressed CSV files containing the class distribution produced by each classifier that serves as input to a later ensemble learning phase.
More details of EI can be found in our Biorxiv preprint:
Full citation:
Yan Chak Li, Linhua Wang, Jeffrey N Law, T M Murali, Gaurav Pandey, Integrating multimodal data through interpretable heterogeneous ensembles, Bioinformatics Advances, Volume 2, Issue 1, 2022, vbac065, https://doi.org/10.1093/bioadv/vbac065
This repository is protected by CC BY-NC 4.0.
This can be done using sdkman (https://sdkman.io/).
python==3.7.4
scikit-learn==0.22
xgboost==1.2.0
numpy==1.19.5
pandas==0.25.3
argparse==1.1
scipy==1.3.1
curl -O -L https://prdownloads.sourceforge.net/weka/weka-3-8-5-azul-zulu-linux.zip
Under the data path, 2 files and a list of feature folders are expected:
-
classifiers.txt This file specifies the list of base classifiers. See the sample_data/classifiers.txt as an example.
-
weka.properties This file specifies the list of weka properties that are parsed to the training/testing of base classifiers. See the sample_data/weka.properties as an example.
-
Folders for feature sets This is a list of folders under the main data path. Each of them originally contains only one file named as data.arff. The .arff files are the input feature matrices and labels for training/testing Weka base classifiers. Indices and labels of .arff files should be aligned across all feature sets.
sample_folder of this repository is an example for reference.
We uploaded the sample data used in the paper to zenodo.
The compressed zip files PFP.zip contains the input data used for EI.
For PFP, since the raw data is very large (around 2139 * 2GB), we uploaded 5 samples of the GO terms which have been transformed into the format for EI. The remaining terms can be generated by the STRING DB (PFP/STRING_csv) & GO annotation files (GO_annotation.tsv) using generate_data.py
For example, you may generate the input data for predicting GO:0000166 by the following command:
python processing_scripts/generate_data.py --outcome GO:0000166
Due to IRB constraints, we are currently unable to publicly share the COVID-19 EHR dataset used in our study. However, we shared the model built based on the dataset for application in covid19-model-built.zip which can load by using load_models.py (more detail here).
Arguments of train_base.py:
--path, -P: Path of the multimodal data
--queue, -Q: LSF queue to submit the job
--node, -N: number of node requested to HPC
--time, -T: number of hours requested to HPC
--memory, -M: memory requsted in MB to HPC
--classpath, -CP: Path of 'weka.jar' (default:'./weka.jar')
--hpc: use HPC cluster or not
--fold, -F: number of cross-validation fold
Option 1: Without access to Minerva, EI can be run sequentially.
python train_base.py --path [data path] --hpc False
Option 2: Run the pipeline in parallel on Minerva HPC
python train_base.py --path [data path] --node [#node] --queue [queue] --time [hour:min] --memory [memory]
Arguments of ensemble.py:
--path, -P: Path of the multimodal data
--fold, -F: cross-validation fold
Run the following command:
python ensemble.py --path [data path]
F-max scores of these models will be printed and written in the performance.csv file and saved to the analysis folder under the data path.
The prediction scores by the ensemble methods will be saved in predictions.csv file in analysis folder under the data path.
Similar to the above step, we will run train_base.py and ensemble.py again, with option --rank True, to train the EI by the whole dataset. All these results will be created in path/model_built folder.
We first generate the local feature ranks (LFR) by the following:
python train_base.py --path [path] --rank True
This step will generate a new folder feature_rank under the data path, which contains a dataset merged with a pseudo test set only for interpretation purposes.
From the path/analysis/performance.csv generated before (--rank=False), we may determine the performance of the ensembles by the Nested-CV setup. We suggest using the best-performing ensemble for EI, eg S.LR, CES, Mean etc. So we may generate the local model rank (LMR) by the following:
python ensemble.py --path [path] --rank True --ens [ensemble algorithm]
After these two steps for calculating LFR and LMR, we may run the ensemble feature ranking by the following:
python ensemble_ranking.py --path [path] --ens [ensemble algorithm]
We may save both local models and EI models for further inference by setting --writeModel True for both train_base.py and ensemble.py
Local models were saved by:
python train_base.py --path [path] --writeModel True
By default, the following command saves all the ensemble models of EI. We may save the specific ensemble model only (e.g. the best-performing ensemble for EI) by specifying --ens option:
python ensemble.py --path [path] --writeModel True --ens [ensemble algorithm, default:all ensemble algorithms]
Loading local models and make base prediction to new dataset (the model_path would be the path/model_built):
python load_models.py --data_path [new dataset path] --model_path [model path] --local_predictor True
We suggest using the best-performing ensemble for EI (eg S.LR, CES, Mean etc.) known from Nested-CV setup. We can use the saved ensemble model to perform integrative prediction, after obtaining the base prediction of new dataset:
python load_models.py --data_path [new dataset path] --model_path [model path] --ens [ensemble model]
After this step, prediction_scores.csv containing predictions of new dataset is generated in data_path/analysis folder.
We used 10 standard binary classification algorithms, such as support vector machine (SVM), random forest (RF) and logistic regression (LR), as implemented in Weka to derive local predictive models from each individual data modality.
Here are the base classifier included in classifier.txt, which are used in train_base.py.
| Base Classifier Name | Weka Class Name |
|---|---|
| AdaBoost | weka.classifiers.meta.AdaBoostM1 |
| Decision Tree | weka.classifiers.trees.J48 |
| Gradient Boosting | weka.classifiers.meta.LogitBoost |
| K-nearest Neighbors | weka.classifiers.lazy.IBk |
| Logistic Regression | weka.classifiers.functions.Logistic -M 100 |
| Voted Perceptron | weka.classifiers.functions.VotedPerceptron |
| Naive Bayes | weka.classifiers.bayes.NaiveBayes |
| Random Forest | weka.classifiers.trees.RandomForest |
| Support Vector Machine | weka.classifiers.functions.SMO -C 1.0E-3 |
| Rule-based classification | weka.classifiers.rules.PART |
We then applied the mean aggregation, ensemble selection method, and stacking to these local models to generate the final EI model.
Here are the meta-classifiers used in stacking, which are used in ensemble.py.
| Meta-classifier Name | Python Class Name | Short Name |
|---|---|---|
| AdaBoost | sklearn.ensemble.AdaBoostClassifier | S.AB |
| Decision Tree | sklearn.tree.DecisionTreeClassifier | S.DT |
| Gradient Boosting | sklearn.ensemble.GradientBoostingClassifier | S.GB |
| K-nearest Neighbors | sklearn.neighbors.KNeighborsClassifier | S.KNN |
| Logistic Regression | sklearn.linear_model.LogisticRegression | S.LR |
| Naive Bayes | sklearn.naive_bayes.GaussianNB | S.NB |
| Random Forest | sklearn.ensemble.RandomForestClassifier | S.RF |
| Support Vector Machine | sklearn.svm.SVC(kernel='linear') | S.SVM |
| XGBoost | xgboost.XGBClassifier | S.XGB |