Lookahead Decoding Development Roadmap

[Viol2000]

Software Quality

• Refactor Code #9
• Simple way to add new model

Implementation

• Support FlashAttention
• Support Sampling
• Support Batch>1
• Lookahead window KV-Cache (May hurt accuracy)
• Verification branch trie

New Models

• Baichuan #11
• QWen #22

[排气扇]

Does this project support the vicuna model?

[Viol2000]

Does this project support the vicuna model?

Vicuna is already supported because it is based on LlamaForCausalLM.

[排气扇]

Thank you for your reply!Do you mean that I can use the following code to observe the acceleration effect of the vicuna model?
USE_LADE=1 python applications/chatbot.py --model_path meta-llama/vicuna-7b-v.13 --debug #no chat, with lookahead
USE_LADE=0 python applications/chatbot.py --model_path meta-llama/vicuna-7b-v.13 --debug #no chat, without lookahead

[Viol2000]

Thank you for your reply!Do you mean that I can use the following code to observe the acceleration effect of the vicuna model? USE_LADE=1 python applications/chatbot.py --model_path meta-llama/vicuna-7b-v.13 --debug #no chat, with lookahead USE_LADE=0 python applications/chatbot.py --model_path meta-llama/vicuna-7b-v.13 --debug #no chat, without lookahead

It should be lmsys/vicuna-7b-v1.3 and yes.

[排气扇]

Got it!Thank you for your reply again!

[LiweiPE]

Hi, Im really interesting in this decoding development. Is there any progress to integrate in Qwen model? thanks.