jongwooko / distillm

Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)

https://arxiv.org/abs/2402.03898

Repository from Github https://github.comjongwooko/distillm

jongwooko/distillm Issues

Dataset link seems invalid now
Closed 6 months ago2
Problem during finetuning qwen2 models
Updated 6 months ago
Download the training/evaluation data
Closed 8 months ago2
gdk mismatch in equation?
Closed 8 months ago
incorrect loss functions based on KLD definition
Closed 9 months ago4
the code of task-specific experiments
Updated 10 months ago
Inconsistnet with your reported scores of teacher models
Closed a year ago12
The metric of GPT-4 Eval
Closed a year ago1
How to download the models after sft
Closed a year ago4
OPTS+=" --kd-ratio 1.0"
Closed a year ago2