FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation

Jingwen Guo, Hong Liu, Shitong Sun, Tianyu Guo, Ming Zhang, and Chenyang Si

ICCV 2023

Existing skeleton-based action recognition methods typically follow a centralized learning paradigm, which can pose privacy concerns when exposing human-related videos. Federated Learning (FL) has attracted much attention due to its outstanding advantages in privacy-preserving. However, directly applying FL approaches to skeleton videos suffers from unstable training. In this paper, we investigate and discover that the heterogeneous human topology graph structure is the crucial factor hindering training stability. To address this issue, we pioneer a novel Federated Skeleton-based Action Recognition (FSAR) paradigm, which enables the construction of a globally generalized model without accessing local sensitive data. Specifically, we introduce an Adaptive Topology Structure (ATS), separating generalization and personalization by learning a domain-invariant topology shared across clients and a domain-specific topology decoupled from global model aggregation. Furthermore, we explore Multi-grain Knowledge Distillation (MKD) to mitigate the discrepancy between clients and the server caused by distinct updating patterns through aligning shallow block-wise motion features. Extensive experiments on multiple datasets demonstrate that FSAR outperforms state-of-the-art FL-based methods while inherently protecting privacy for skeleton-based action recognition.

The local clients are optimized with our proposed Adaptive Topology Structure (ATS) and Multi-grain Knowledge Distillation (MKD) modules on private data and then perform the client-server collaborative learning iteratively: (i) clients train local models; (ii) clients upload parameters to server; (iii) server aggregates model parameters; (iv) clients download the aggregated models. Moreover, the ATS module extracts the intrinsic structure information of heterogeneous skeleton data, and the MKD module bridges the divergence between the clients and the server.

Requirements

Installation

# Install python environment
$ conda create -n fsar python=3.8.2
$ conda activate fsar

# Install PyTorch
$ pip install torch==1.4.0

# Install other python libraries
$ pip install -r requirements.txt

Dataset setup

For all the datasets, be sure to read and follow their license agreements, and cite them accordingly. The datasets we used are as follows:

Train the model

To train the model under the federated-by-dataset or federated-by-class scenarios:

cd fsar/application/fedhar/
CUDA_VISIBLE_DEVICES=0 python main.py

Acknowledgement

Our code is extended from the following repositories. We thank the authors for releasing the codes.

Licence

This project is licensed under the terms of the MIT license.

DivyaGuo / FSAR