knowledge-distillation

There are 55 repositories under knowledge-distillation topic.

PaddlePaddle / PaddleClas
A treasure chest for visual classification and recognition powered by PaddlePaddle
image-classification knowledge-distillation autoaugment randaugment gridmask cutmix deit repvgg swin-transformer pretrained-models image-retrieval-system image-recognition product-recognition fastdeploy
Language:Python 5750
dkozlov / awesome-knowledge-distillation
Awesome Knowledge Distillation
knowledge-distillation knowledge-transfer teacher-student co-training distillation distillation-model model-distillation knowldge-distillation kd model-compression deep-learning
3762
huawei-noah / Pretrained-Language-Model
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
knowledge-distillation model-compression quantization pretrained-models large-scale-distributed
Language:Python 3143
jingyi0000 / VLM_survey
Collection of AWESOME vision-language models for vision tasks
clip computer-vision deep-learning knowledge-distillation multi-modal-model survey transfer-learning vision-language-model
2992
IDEA-Research / DWPose
"Effective Whole-body Pose Estimation with Two-stages Distillation" (ICCV 2023, CV4Metaverse Workshop)
controlnet knowledge-distillation pose-estimation stable-diffusion-webui
Language:Python 2599
intel / neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime
auto-tuning awq fp4 gptq int4 int8 knowledge-distillation large-language-models low-precision mxformat post-training-quantization pruning quantization quantization-aware-training smoothquant sparsegpt sparsity
Language:Python 2520
EasyNLP
alibaba / EasyNLP
EasyNLP: A Comprehensive and Easy-to-use NLP Toolkit
transformers bert nlp pretrained-models deep-learning pytorch fewshot-learning knowledge-distillation knowledge-pretraining text-image-retrieval text-to-image-synthesis machine-learning text-classification transfer-learning
Language:Python 2172
haitongli / knowledge-distillation-pytorch
A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility
pytorch knowledge-distillation deep-neural-networks cifar10 model-compression dark-knowledge computer-vision
Language:Python 1970
horseee / Awesome-Efficient-LLM
A curated list for Efficient Large Language Models
compression efficient-llm knowledge-distillation language-model llm llm-compression model-quantization pruning-algorithms
Language:Python 1892
microsoft / Cream
This is a collection of our NAS and Vision Transformer work.
automl efficiency knowledge-distillation nas rpe vision-transformer vit-compression
Language:Python 1809
AberHu / Knowledge-Distillation-Zoo
Pytorch implementation of various Knowledge Distillation (KD) methods.
kd-methods knowledge-transfer knowledge-distillation kd teacher-student model-compression distillation
Language:Python 1727
open-mmlab / mmrazor
OpenMMLab Model Compression Toolbox and Benchmark.
nas pruning knowledge-distillation spos darts autoslim detection segmentation classification pytorch quantization
Language:Python 1640
torchdistill
yoshitomo-matsubara / torchdistill
A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆26 knowledge distillation methods presented at CVPR, ICLR, ECCV, NeurIPS, ICCV, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.
knowledge-distillation pytorch image-classification imagenet object-detection coco semantic-segmentation cifar10 cifar100 colab-notebook google-colab pascal-voc nlp natural-language-processing transformer glue amazon-sagemaker-lab pytorch-ecosystem text-classification
Language:Python 1568
szagoruyko / attention-transfer
Improving Convolutional Networks via Attention Transfer (ICLR 2017)
attention deep-learning knowledge-distillation pytorch
Language:Jupyter Notebook 1462
microsoft / NeuronBlocks
NLP DNN Toolkit - Building Your NLP DNN Models Like Playing Lego
artificial-intelligence deep-learning dnn knowledge-distillation model-compression natural-language-processing pytorch qna question-answering sequence-labeling text-classification text-matching
Language:Python 1452
lxztju / pytorch_classification
利用pytorch实现图像分类的一个完整的代码，训练，预测，TTA，模型融合，模型部署，cnn提取特征，svm或者随机森林等进行分类，模型蒸馏，一个完整的代码
cnn densenet deployment flask image-classification knn knowledge-distillation label-smoothing pytorch random-forest resnet resnext svm
Language:Jupyter Notebook 1451
robert-mcdermott / ai-knowledge-graph
AI Powered Knowledge Graph Generator
artificial-intelligence knowledge-distillation knowledge-graph llm networkx pyvis visualization
Language:Python 1355
huawei-noah / Efficient-Computing
Efficient computing methods developed by Huawei Noah's Ark Lab
knowledge-distillation model-compression binary-neural-networks pruning quantization self-supervised
Language:Jupyter Notebook 1300
Tebmer / Awesome-Knowledge-Distillation-of-LLMs
This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.
data-augmentation instruction-following kd knowledge-distillation large-language-model llm self-training survey compression data-synthesis feedback multi-modal self-distillation alignment supervised-finetuning
1202
MingSun-Tse / Efficient-Deep-Learning
Collection of recent methods on (deep) neural network compression and acceleration.
model-compression network-pruning knowledge-distillation deep-learning deep-neural-networks efficient-deep-learning
951
megvii-research / mdistiller
The official implementation of [CVPR2022] Decoupled Knowledge Distillation https://arxiv.org/abs/2203.08679 and [ICCV2023] DOT: A Distillation-Oriented Trainer https://openaccess.thecvf.com/content/ICCV2023/papers/Zhao_DOT_A_Distillation-Oriented_Trainer_ICCV_2023_paper.pdf
cifar coco computer-vision cvpr2022 deep-learning iccv2023 imagenet knowledge-distillation pytorch
Language:Python 876
EasyTransfer
alibaba / EasyTransfer
EasyTransfer is designed to make the development of transfer learning in NLP applications easier.
bert nlp-applications knowledge-distillation transfer-learning
Language:Python 860
lhyfst / knowledge-distillation-papers
knowledge distillation papers
knowledge-distillation model-compression paper dark-knowledge reading-list
762
mammoth
aimagelab / mammoth
An Extendible (General) Continual Learning Framework based on Pytorch - official codebase of Dark Experience for General Continual Learning
pytorch continual-learning neurips2020 knowledge-distillation deep-learning der experience-replay dark-experience-replay
Language:Python 749
SforAiDl / KD_Lib
A Pytorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.
knowledge-distillation model-compression pruning quantization pytorch deep-learning-library machine-learning data-science benchmarking algorithm-implementations
Language:Python 647
segmind / distill-sd
Segmind Distilled diffusion
distillation inference knowledge-distillation stable-diffusion
Language:Python 612
Picsart-AI-Research / MI-GAN
[ICCV 2023] MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices
image-inpainting knowledge-distillation object-removal mi-gan mobile-inpainting re-parameterization
Language:Python 601
yuanli2333 / Teacher-free-Knowledge-Distillation
Knowledge Distillation: CVPR2020 Oral, Revisiting Knowledge Distillation via Label Smoothing Regularization
knowledge-distillation label-smoothing paper-implementations pytorch teacher-free
Language:Python 584
chaitjo / efficient-gnns
Code and resources on scalable and efficient Graph Neural Networks (TNNLS 2023)
graph-neural-networks graph-representation-learning graph-convolutional-networks gnn efficient-deep-learning efficient-neural-networks scalable-deep-learning contrastive-learning knowledge-distillation
Language:Python 536
yanx27 / 2DPASS
2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds (ECCV 2022) :fire:
nuscenes semantic-segmentation semantickitti lidar-point-cloud deep-learning knowledge-distillation pytorch
Language:Python 448
1duo / awesome-ai-infrastructures
Infrastructures™ for Machine Learning Training/Inference in Production.
artificial-intelligence machine-learning deep-learning machine-learning-systems awesome-list deep-learning-framework kubernetes apache-spark apache-arrow apache-mesos pruning quantization knowledge-distillation model-compression federated-learning
428
clovaai / overhaul-distillation
Official PyTorch implementation of "A Comprehensive Overhaul of Feature Distillation" (ICCV 2019)
knowledge-distillation knowledge-transfer teacher-student network-compression iccv2019
Language:Python 418
lenscloth / RKD
Official pytorch Implementation of Relational Knowledge Distillation, CVPR 2019
computer-vision deep-learning deep-neural-networks knowledge-distillation metric-learning
Language:Python 405
ilovepose / fast-human-pose-estimation.pytorch
Official pytorch Code for CVPR2019 paper "Fast Human Pose Estimation" https://arxiv.org/abs/1811.05419
human-pose-estimation deep-learning coco-keypoints-detection mpii-dataset mscoco-keypoint fast-pose-distillation knowledge-distillation
Language:Cuda 397
yzhao062 / SUOD
(MLSys' 21) An Acceleration System for Large-scare Unsupervised Heterogeneous Outlier Detection (Anomaly Detection)
anomaly-detection data-mining distributed-systems knowledge-distillation machine-learning machine-learning-algorithms machine-learning-library outlier-detection python
Language:Python 391
HikariTJU / LD
Localization Distillation for Object Detection (CVPR 2022, TPAMI 2023)
detection knowledge-distillation ld object-detection pytorch deep-learning
Language:Python 385

knowledge-distillation

PaddlePaddle / PaddleClas

dkozlov / awesome-knowledge-distillation

huawei-noah / Pretrained-Language-Model

jingyi0000 / VLM_survey

IDEA-Research / DWPose

intel / neural-compressor

alibaba / EasyNLP

haitongli / knowledge-distillation-pytorch

horseee / Awesome-Efficient-LLM

microsoft / Cream

AberHu / Knowledge-Distillation-Zoo

open-mmlab / mmrazor

yoshitomo-matsubara / torchdistill

szagoruyko / attention-transfer

microsoft / NeuronBlocks

lxztju / pytorch_classification

robert-mcdermott / ai-knowledge-graph

huawei-noah / Efficient-Computing

Tebmer / Awesome-Knowledge-Distillation-of-LLMs

MingSun-Tse / Efficient-Deep-Learning

megvii-research / mdistiller

alibaba / EasyTransfer

lhyfst / knowledge-distillation-papers

aimagelab / mammoth

SforAiDl / KD_Lib

segmind / distill-sd

Picsart-AI-Research / MI-GAN

yuanli2333 / Teacher-free-Knowledge-Distillation

chaitjo / efficient-gnns

yanx27 / 2DPASS

1duo / awesome-ai-infrastructures

clovaai / overhaul-distillation

lenscloth / RKD

ilovepose / fast-human-pose-estimation.pytorch

yzhao062 / SUOD

HikariTJU / LD