Giters

Tebmer / Awesome-Knowledge-Distillation-of-LLMs

This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Missing paper (Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs)

Nicolas-BZRD opened this issue · comments

commented

Hy guys, thanks for this great initiative. We recently release a new papers paving the way to a new method to distil any LLM to any other. I thing it could be very useful to add the paper on the list. Feel free to ask me any questions if needed.

Paper (Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs): https://arxiv.org/abs/2402.12030

commented

Great work! Thanks. We have added it :) and will update paper in a future version.