A curated list of awesome papers on dataset distillation and related applications, inspired by awesome-computer-vision.
Dataset distillation is the task of synthesizing a small dataset such that models trained on it achieve high performance on the original large dataset. A dataset distillation algorithm takes as input a large real dataset to be distilled (training set), and outputs a small synthetic distilled dataset, which is evaluated via testing models trained on this distilled dataset on a separate real dataset (validation/test set). A good small distilled dataset is not only useful in dataset understanding, but has various applications (e.g., continual learning, privacy, neural architecture search, etc.). This task was first introduced in the 2018 paper Dataset Distillation [Tongzhou Wang et al., '18], along with a proposed algorithm using backpropagation through optimization steps.
In recent years (2019-now), dataset distillation has gained increasing attention in the research community, across many institutes and labs. More papers are now being published each year. These wonderful researches have been constantly improving dataset distillation and exploring its various variants and applications.
This project is curated and maintained by Guang Li, Bo Zhao, and Tongzhou Wang.
- ๐ Project Page
Code- ๐
bibtex
Media Coverage
Acknowledgments
- Dataset Distillation (Tongzhou Wang et al., 2018) ๐ ๐
- Gradient-Based Hyperparameter Optimization Through Reversible Learning (Dougal Maclaurin et al., ICML 2015) ๐
- Dataset Condensation with Gradient Matching (Bo Zhao et al., ICLR 2021) ๐
- Dataset Condensation with Differentiable Siamese Augmentation (Bo Zhao et al., ICML 2021) ๐
- Dataset Distillation by Matching Training Trajectories (George Cazenavette et al., CVPR 2022) ๐ ๐
- Dataset Condensation with Contrastive Signals (Saehyung Lee et al., ICML 2022) ๐
- Delving into Effective Gradient Matching for Dataset Condensation (Zixuan Jiang et al., 2022) ๐
- Dataset Condensation with Distribution Matching (Bo Zhao et al., 2021) ๐
- CAFE: Learning to Condense Dataset by Aligning Features (Kai Wang & Bo Zhao et al., CVPR 2022) ๐
- Dataset Factorization for Condensation (Songhua Liu et al., NeurIPS 2022) ๐
- Optimizing Millions of Hyperparameters by Implicit Differentiation (Jonathan Lorraine et al., AISTATS 2020) ๐
- Dataset Meta-Learning from Kernel Ridge-Regression (Timothy Nguyen et al., ICLR 2021) ๐
- Dataset Distillation with Infinitely Wide Convolutional Networks (Timothy Nguyen et al., NeurIPS 2021) ๐
- On Implicit Bias in Overparameterized Bilevel Optimization (Paul Vicol et al., ICML 2022) ๐
- Dataset Distillation using Neural Feature Regression (Yongchao Zhou et al., NeurIPS 2022) ๐ ๐
- Efficient Dataset Distillation using Random Feature Approximation (Noel Loo et al., NeurIPS 2022) ๐
- Synthesizing Informative Training Samples with GAN (Bo Zhao et al., 2022) ๐
- Dataset Condensation via Efficient Synthetic-Data Parameterization (Jang-Hyun Kim et al., ICML 2022) ๐
- Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks (Zhiwei Deng et al., NeurIPS 2022) ๐
- PRANC: Pseudo RAndom Networks for Compacting deep models (Parsa Nooralinejad et al., 2022) ๐
- Dataset Condensation with Latent Space Knowledge Factorization and Sharing (Hae Beom Lee et al., 2022) ๐
- Flexible Dataset Distillation: Learn Labels Instead of Images (Ondrej Bohdal et al., NeurIPS 2020 Workshop) ๐
- Soft-Label Dataset Distillation and Text Dataset Distillation (Ilia Sucholutsky et al., IJCNN 2021) ๐
- DC-BENCH: Dataset Condensation Benchmark (Justin Cui et al., NeurIPS 2022) ๐
- Reducing Catastrophic Forgetting with Learning on Synthetic Data (Wojciech Masarczyk et al., CVPR 2020 Workshop) ๐
- Condensed Composite Memory Continual Learning (Felix Wiewel et al., IJCNN 2021) ๐
- Distilled Replay: Overcoming Forgetting through Synthetic Samples (Andrea Rosasco et al., 2021) ๐
- Sample Condensation in Online Continual Learning (Mattia Sangermano et al., IJCNN 2022) ๐
- SecDD: Efficient and Secure Method for Remotely Training Neural Networks (Ilia Sucholutsky et al., AAAI 2021 Student Abstract) ๐
- Privacy for Free: How does Dataset Condensation Help Privacy? (Tian Dong et al., ICML 2022) ๐
- Can We Achieve Robustness from Data Alone? (Nikolaos Tsilivis et al., 2022) ๐
- Soft-Label Anonymous Gastric X-ray Image Distillation (Guang Li et al., ICIP 2020) ๐
- Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing (Guang Li et al., CMPB 2022) ๐
- Dataset Distillation for Medical Dataset Sharing (Guang Li et al., 2022) ๐
- Dataset Distillation using Parameter Pruning (Guang Li et al., 2022) ๐
- Federated Learning via Synthetic Data (Jack Goetz et al., 2020) ๐
- Distilled One-Shot Federated Learning (Yanlin Zhou et al., 2020) ๐
- FedSynth: Gradient Compression via Synthetic Data in Federated Learning (Shengyuan Hu et al., 2022) ๐
- FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning (Yuanhao Xiong et al., 2022) ๐
- Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments (Rui Song et al., 2022) ๐
- Graph Condensation for Graph Neural Networks (Wei Jin et al., ICLR 2022) ๐
- Condensing Graphs via One-Step Gradient Matching (Wei Jin et al., KDD 2022) ๐
- Graph Condensation via Receptive Field Distribution Matching (Mengyang Liu et al., 2022) ๐
- Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data (Felipe Petroski Such et al., ICML 2020) ๐
- Wearable ImageNet: Synthesizing Tileable Textures via Dataset Distillation (George Cazenavette et al., CVPR 2022 Workshop) ๐ ๐
- Knowledge Condensation Distillation (Chenxin Li et al., ECCV 2022) ๐
- Infinite Recommendation Networks: A Data-Centric Approach (Noveen Sachdeva et al., NeurIPS 2022) ๐
- Bidirectional Learning for Offline Infinite-width Model-based Optimization (Can Chen et al., NeurIPS 2022) ๐
- Data Distillation for Text Classification (Yongqi Li et al., 2021) ๐
- Beginning of Awesome-Dataset-Distillation
- Most Popular AI Research Aug 2022
- ไธไธช้กน็ฎๅธฎไฝ ไบ่งฃๆฐๆฎ้่ธ้ฆDataset Distillation
We would like to thank Nikolaos Tsilivis, Wei Jin, Yongchao Zhou, Noveen Sachdeva, Can Chen and Guangxiang Zhao for their valuable suggestions and contributions.