Zhiqin Yang*, Yonggang Zhang*, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, Bo Han
Keywords: Feature Distillation, Data Heterogeneity, Federated Learning
Abstract: Federated learning (FL) typically faces data heterogeneity, i.e., distribution shifting among clients. Sharing clientsβ information has shown great potentiality in mitigating data heterogeneity, yet incurs a dilemma in preserving privacy and promoting model performance. To alleviate the dilemma, we raise a fundamental question: Is it possible to share partial features in the data to tackle data heterogeneity? In this work, we give an affirmative answer to this question by proposing a novel approach called Federated Feature distillation (FedFed). Specifically, FedFed partitions data into performance-sensitive features (i.e., greatly contributing to model performance) and performance-robust features (i.e., limitedly contributing to model performance). The performance-sensitive features are globally shared to mitigate data heterogeneity, while the performance-robust features are kept locally. FedFed enables clients to train models over local and shared data. Comprehensive experiments demonstrate the efficacy of FedFed in promoting model performance
- π¬ Dependencies and installation
- π₯³ How to run:
- π Extension based on our code
- πΉ Experimental results
- π Citation
- π License
- π₯° Acknowledgement
- βοΈ Contact
π If FedFed is helpful to you, please star this repo. Thanks! π€
Before running or modifing the code, you need to:
-
Make sure Anaconda or Miniconda is installed.
-
Clone this repo to your machine.
# git clone this repository git clone https://github.com/visitworld123/FedFed cd FedFed # create new anaconda env conda create -n FedFed python=3.8 -y conda activate FedFed
-
required environment in
environment.yaml
# install environment dependencies conda env create -f environment.yaml
-
required packages in
requirements.txt
# install python dependencies pip install -r requirements.txt
python main.py --config_file config.yaml
In config.yaml, you can change the hyper-parameters and some settings. In our experiments, we use wandb as the default tool to record the running metrics like Acc and Loss. If you don't want to use wandb, you can set wandb_record = False
. And if you want to use tensorboard to record, you can set record_tool = 'tensorboard
.
Please refer to the ./configs/default.py
file for the list of parameters. We list some main parameters and its meaning to help quick start.
You can set these parameters by modify default.py
or a new yaml
file and use it by --config_file xxx.yaml
.
--client_num_in_total
: The total number of clients in FL system, you can set 10 and 100 to reproduce our experiments.
--client_num_per_round
: The number of sampled clients every round, also called sampling rate.
--partition_alpha
: Our main results based on LDA partition method, you can set 0.1 and 0.05.
--global_epochs_per_round
: Local epochs for every client.
--comm_round
: Global Communication round.
βββ algorithms_standalone
β βββ basePS
β β βββ aggregator.py # Details of federated aggregation, one can extend based on this
β β βββ basePSmanager.py # Details of federated manager, one can extend based on this
β β βββ client.py # Details of federated clients, one can extend based on this
β βββ fedavg # FedAvg folder, this folder also including FedProx and SCAFFOLD
β β βββ FedAVGManager.py # FedAvg manager, based on basePS
β β βββ aggregator.py # FedAvg aggregator, based on basePS
β β βββ client.py # FedAvg clients, based on basePS
β βββ fednova
β βββ FedNovaManager.py # FedNova manager, based on basePS
β βββ aggregator.py # FedNova aggregator, based on basePS
β βββ client.py # FedNova clients, based on basePS
βββ configs
β βββ config.py
β βββ default.py # Manage parameters by default
βββ data_preprocessing # Process dataset into heter distribution across clients
β βββ loader.py
β βββ build.py
βββ loss_fn # Some loss functions can be choose
βββ model
β βββ FL_VAE.py # Model for feature distillation
β βββ build.py
β βββ cv # Local classifier model of federated learning in CV
βββ optim # Build optimizer for local client
β βββ build.py
βββ trainers
β βββ normal_trainer.py # Details of local training
βββ utils
βββ main.py
βββ config.yaml
In the directory of /algorithms_standalone/basePS
. It contains three file: aggregator.py, basePSmanager.py, and client.py.
These three files have some basic functions that in FL. If you want to implement your own algorithm based on our code. Your new aggregator, client and manager can inherit from these basic classes.
If our work is useful for your research, please consider citing:
@inproceedings{fedfed,
title = {FedFed: Feature Distillation against Data Heterogeneity in Federated Learning},
author={Yang, Zhiqin and Zhang, Yonggang and Zheng, Yu and Tian, Xinmei and Peng, Hao and Liu, Tongliang and Han, Bo},
booktitle = {Advances in Neural Information Processing Systems},
year = {2023}
}
This project is licensed under MIT. Redistribution and use should follow this license.
This project is partly based on VHL and CD-VAE.
This Readme is inspired by codeformer π.
If you have any questions, please feel free to reach me out at yangzqccc@buaa.edu.cn
.