There are 2 repositories under large-scale-distributed topic.
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
Notes on Large scale distributed systems