LibMTL

LibMTL is an open-source library built on PyTorch for Multi-Task Learning (MTL). See the latest documentation for detailed introductions and API instructions.

⭐ Star us on GitHub — it motivates us a lot!

Table of Content

Features
Overall Framework
Supported Algorithms
Installation
Quick Start
- Download Dataset
- Run a Model
Citation
Contributors
Contact Us
Acknowledgements
License

Features

Unified: LibMTL provides a unified code base to implement and a consistent evaluation procedure including data processing, metric objectives, and hyper-parameters on several representative MTL benchmark datasets, which allows quantitative, fair, and consistent comparisons between different MTL algorithms.
Comprehensive: LibMTL supports 84 MTL models combined by 7 architectures and 12 loss weighting strategies. Meanwhile, LibMTL provides a fair comparison on 3 computer vision datasets.
Extensible: LibMTL follows the modular design principles, which allows users to flexibly and conveniently add customized components or make personalized modifications. Therefore, users can easily and fast develop novel loss weighting strategies and architectures or apply the existing MTL algorithms to new application scenarios with the support of LibMTL.

Overall Framework

Config Module: Responsible for all the configuration parameters involved in the running framework, including the parameters of optimizer and learning rate scheduler, the hyper-parameters of MTL model, training configuration like batch size, total epoch, random seed and so on.
Dataloaders Module: Responsible for data pre-processing and loading.
Model Module: Responsible for inheriting classes architecture and weighting and instantiating a MTL model. Note that the architecture and the weighting strategy determine the forward and backward processes of the MTL model, respectively.
Losses Module: Responsible for computing the loss for each task.
Metrics Module: Responsible for evaluating the MTL model and calculating the metric scores for each task.

Supported Algorithms

LibMTL currently supports the following algorithms:

12 loss weighting strategies.

Weighting Strategy	Venues	Comments
Equally Weighting (EW)	-	Implemented by us
Gradient Normalization (GradNorm)	ICML 2018	Implemented by us
Uncertainty Weights (UW)	CVPR 2018	Implemented by us
MGDA	NeurIPS 2018	Referenced from official PyTorch implementation
Dynamic Weight Average (DWA)	CVPR 2019	Referenced from official PyTorch implementation
Geometric Loss Strategy (GLS)	CVPR 2019 workshop	Implemented by us
Projecting Conflicting Gradient (PCGrad)	NeurIPS 2020	Implemented by us
Gradient sign Dropout (GradDrop)	NeurIPS 2020	Implemented by us
Impartial Multi-Task Learning (IMTL)	ICLR 2021	Implemented by us
Gradient Vaccine (GradVac)	ICLR 2021 Spotlight	Implemented by us
Conflict-Averse Gradient descent (CAGrad)	NeurIPS 2021	Referenced from official PyTorch implementation
Random Loss Weighting (RLW)	arXiv	Implemented by us

7 architectures.

Architecture	Venues	Comments
Hard Parameter Sharing (HPS)	ICML 1993	Implemented by us
Cross-stitch Networks (Cross_stitch)	CVPR 2016	Implemented by us
Multi-gate Mixture-of-Experts (MMoE)	KDD 2018	Implemented by us
Multi-Task Attention Network (MTAN)	CVPR 2019	Referenced from official PyTorch implementation
Customized Gate Control (CGC)	ACM RecSys 2020 Best Paper	Implemented by us
Progressive Layered Extraction (PLE)	ACM RecSys 2020 Best Paper	Implemented by us
DSelect-k	NeurIPS 2021	Referenced from official TensorFlow implementation

84 combinations of different architectures and loss weighting strategies.

Installation

The simplest way to install LibMTL is using pip.

pip install -U LibMTL

More details about environment configuration is represented in Docs.

Quick Start

We use the NYUv2 dataset as an example to show how to use LibMTL.

Download Dataset

The NYUv2 dataset we used is pre-processed by mtan. You can download this dataset here.

Run a Model

The complete training code for the NYUv2 dataset is provided in examples/nyu. The file train_nyu.py is the main file for training on the NYUv2 dataset.

You can find the command-line arguments by running the following command.

python train_nyu.py -h

For instance, running the following command will train a MTL model with EW and HPS on NYUv2 dataset.

python train_nyu.py --weighting EW --arch HPS --dataset_path /path/to/nyuv2 --gpu_id 0 --scheduler step

More details is represented in Docs.

Citation

If you find LibMTL useful for your research or development, please cite the following:

@misc{LibMTL,
 author = {Baijiong Lin and Yu Zhang},
 title = {LibMTL: A PyTorch Library for Multi-Task Learning},
 year = {2021},
 publisher = {GitHub},
 journal = {GitHub repository},
 howpublished = {\url{https://github.com/median-research-group/LibMTL}}
}

Contributors

LibMTL is developed and maintained by Baijiong Lin and Yu Zhang.

Contact Us

If you have any question or suggestion, please feel free to contact us by raising an issue or sending an email to bj.lin.email@gmail.com.

Acknowledgements

We would like to thank the authors that release the public repositories (listed alphabetically): CAGrad, dselect_k_moe, MultiObjectiveOptimization, and mtan.

License

LibMTL is released under the MIT license.

Allensmile / LibMTL