hao-ai-lab / LookaheadDecoding

[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

https://arxiv.org/abs/2402.02057

hao-ai-lab/LookaheadDecoding Issues

Tensor parallel
Closed 3 months ago1
Yi Models support
Updated 4 months ago1
How to optimize the scene with relatively short output?
Updated 4 months ago
How to use LADE in single-node multi-process way?
Updated 4 months ago2
question on verification beginning pos
Updated 4 months ago1
about verification branch
Updated 4 months ago1
Bug in Greedy Search
Updated 4 months ago
TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'lookahead'
Updated 5 months ago5
Cannot use save_log and log_history
Updated 5 months ago1
[BUG Report] jacobi_greedy_search_multilevel function bug
Updated 6 months ago
Do lookahead and repetition_penalty conflict?
Updated 6 months ago
BUG Report
Updated 6 months ago
Why jacobi_sample_multilevel() fill window with argmax instead of also using sampling?
Updated 6 months ago
Lookahead Decoding Development Roadmap
Updated 7 months ago6
Adaptive LEVEL/GUESS size?
Updated 7 months ago
Why using Jacobi decoding? What are the advantages besides the fact that Jacobi decoding can reduce some of the decode steps？
Closed 8 months ago19
Related work: Prompt lookup decoding
Updated 8 months ago7
specific model
Updated 8 months ago13
Is it similar to ProphetNet, ProphetNet-Ads and BANG?
Closed 8 months ago2
Is it the same as ProphetNet and ProphetNet-Ads?
Closed 8 months ago
Run time error with `lade.config_lade(LEVEL=2)`
Closed 8 months ago3
How to generate the n-grams - which to keep, which to discard?
Updated 8 months ago27
Undefined symbols in lade/decoding.py
Closed 8 months ago1
Multiple GPUS
Updated 8 months ago9
Compatibility with Flash Attention 2
Closed 9 months ago2
Questions on combined attention mask structure for Jacobi iteration
Updated 9 months ago23
Qs on Understanding Lookahead and Jacobi
Closed 9 months ago7
Incompatible with LlamaSdpaAttention in transformers v4.36
Closed 9 months ago1
Can I get a MT-bench evaluation code for reproduction of acceleration?
Updated 9 months ago11
TypeError: 'NoneType' object is not subscriptable at line 427 of decoding.py
Closed 9 months ago5
Any plan to support other LLMs, and integrated into huggingface?
Closed 9 months ago2
Does Lade support topp/topk/temperature sampleing?
Updated 9 months ago2
Why the tup needs to be added to the tail of token_map when it found in the token_map?
Closed 10 months ago2
Questions on the attention mask, and whether to accept the last element of guess_results when all guess_tokens are accepted
Updated 10 months ago6
Can lade accelerate T5?
Closed 10 months ago1
question about attention patterns
Closed 10 months ago6
Does the module support fine-tuned Llama2?
Closed 10 months ago4
Support Baichuan models
Closed 10 months ago1
Can't run minimal.py on A100
Closed 10 months ago3
NameError: name 'F' is not defined
Closed 10 months ago1
The meaning of the CHAT option in decoding
Closed 10 months ago1
The inference results are inconsistent with Huggingface.
Updated 10 months ago6
The Jacobi method and its corresponding code
Closed 10 months ago2
Benchmarks comparing with Medusa
Updated 10 months ago
amazing work!! any plan support for chatglm or Qwen model?
Updated 10 months ago1
No speed up
Updated 10 months ago5
Support for GPTQ?
Updated 10 months ago4
Support for bnb nf4 quant?
Updated 10 months ago1
Any analysis on the impact on accuracy
Closed 10 months ago2
How about batching throughput and energy consumption
Updated 10 months ago2