Hivemind: decentralized deep learning in PyTorch
Hivemind is a PyTorch library for decentralized deep learning across the Internet. Its intended usage is training one large model on hundreds of computers from different universities, companies, and volunteers.
Live Demo
Check out our NeurIPS 2021 demonstration "Training Transformers Together" to see hivemind in action, join an ongoing collaborative experiment, and learn more about the technologies behind it!
Key Features
- Distributed training without a master node: Distributed Hash Table allows connecting computers in a decentralized network.
- Fault-tolerant backpropagation: forward and backward passes succeed even if some nodes are unresponsive or take too long to respond.
- Decentralized parameter averaging: iteratively aggregate updates from multiple workers without the need to synchronize across the entire network (paper).
- Train neural networks of arbitrary size: parts of their layers are distributed across the participants with the Decentralized Mixture-of-Experts (paper).
To learn more about the ideas behind this library, see the full list of our papers below.
Installation
Before installing, make sure that your environment has Python 3.7+ and PyTorch 1.6.0 or newer. They can be installed either natively or with Anaconda.
You can get the latest release with pip or build hivemind from source.
With pip
If your versions of Python and PyTorch match the requirements, you can install hivemind from pip:
pip install hivemind
From source
To install hivemind from source, simply run the following:
git clone https://github.com/learning-at-home/hivemind.git
cd hivemind
pip install .
If you would like to verify that your installation is working properly, you can install with pip install .[dev]
instead. Then, you can run the tests with pytest tests/.
By default, hivemind uses the precompiled binary of
the go-libp2p-daemon library. If you face compatibility issues
or want to build the binary yourself, you can recompile it by running pip install . --global-option="--buildgo".
Before running the compilation, please ensure that your machine has a recent version
of Go toolchain (1.15 or 1.16 are supported).
System requirements
- Linux is the default OS for which hivemind is developed and tested. We recommend Ubuntu 18.04+ (64-bit), but other 64-bit distros should work as well. Legacy 32-bit is not recommended.
- macOS 10.x can run hivemind using Docker. We recommend using our Docker image.
- Windows 10+ (experimental) can run hivemind using WSL. You can configure WSL to use GPU by following sections 1–3 of this guide by NVIDIA. After that, you can simply follow the instructions above to install with pip or from source.
Documentation
- The quickstart tutorial walks through installation and a training a simple neural network with several peers.
- examples/albert contains the starter kit and instructions for training a Transformer masked language model collaboratively.
- The Mixture-of-Experts tutorial covers the usage of Decentralized Mixture-of-Experts layers.
- API reference and additional tutorials are available at learning-at-home.readthedocs.io
If you have any questions about installing and using hivemind, feel free to ask them in our Discord chat or file an issue.
Contributing
Hivemind is currently at the active development stage, and we welcome all contributions. Everything, from bug fixes and documentation improvements to entirely new features, is appreciated.
If you want to contribute to hivemind but don't know where to start, take a look at the unresolved issues. Open a new issue or join our chat room in case you want to discuss new functionality or report a possible bug. Bug fixes are always welcome, but new features should be preferably discussed with maintainers beforehand.
If you want to start contributing to the source code of hivemind, please see the contributing guidelines first. To learn more about other ways to contribute, read our guide.
Citation
If you found hivemind or its underlying algorithms useful for your research, please cite the following source:
@misc{hivemind,
author = {Learning{@}home team},
title = {{H}ivemind: a {L}ibrary for {D}ecentralized {D}eep {L}earning},
year = 2020,
howpublished = {\url{https://github.com/learning-at-home/hivemind}},
}Also, you can cite the paper that inspired the creation of this library (prototype implementation of hivemind available at mryab/learning-at-home):
@inproceedings{ryabinin2020crowdsourced,
author = {Ryabinin, Max and Gusev, Anton},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
pages = {3659--3672},
publisher = {Curran Associates, Inc.},
title = {Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts},
url = {https://proceedings.neurips.cc/paper/2020/file/25ddc0f8c9d3e22e03d3076f98d83cb2-Paper.pdf},
volume = {33},
year = {2020}
}Additional publications
"Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices"
@misc{ryabinin2021moshpit,
title = {Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices},
author = {Max Ryabinin and Eduard Gorbunov and Vsevolod Plokhotnyuk and Gennady Pekhimenko},
year = {2021},
eprint = {2103.03239},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}"Distributed Deep Learning in Open Collaborations"
@misc{diskin2021distributed,
title = {Distributed Deep Learning in Open Collaborations},
author = {Michael Diskin and Alexey Bukhtiyarov and Max Ryabinin and Lucile Saulnier and Quentin Lhoest and Anton Sinitsin and Dmitry Popov and Dmitry Pyrkin and Maxim Kashirin and Alexander Borzunov and Albert Villanova del Moral and Denis Mazur and Ilia Kobelev and Yacine Jernite and Thomas Wolf and Gennady Pekhimenko},
year = {2021},
eprint = {2106.10207},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}"Secure Distributed Training at Scale"
@misc{gorbunov2021secure,
title = {Secure Distributed Training at Scale},
author = {Eduard Gorbunov and Alexander Borzunov and Michael Diskin and Max Ryabinin},
year = {2021},
eprint = {2106.11257},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}We also maintain a list of related projects and acknowledgements.