pytorch-labs / gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

pytorch-labs/gpt-fast Issues

Has anyone run this code with bs>1 and speculatively?
Updated 12 days ago
Mistake in 191 line if is_speculative=True generate.py ?
Updated 13 days ago1
Error with meta-llama/Llama-3.2-1B
Updated 19 days ago2
Error with stories15M and stories110M
Updated 19 days ago8
The Actual Throughput of int8 Quantization is Significantly Lower than Baseline on A100
Closed 19 days ago4
Request for Smaller Model Options (~1B Parameters)
Updated 19 days ago
Int4 perplexity
Closed a month ago2
Missing Keys in state_dict
Updated 2 months ago2
int4/int4-gptq support in Mixtral 8x7B
Closed 2 months ago4
Tensor Parallel Inside notebook
Updated 2 months ago3
Hard-coded Llama-3 model name pattern matching breaks scripts/convert_hf_checkpoint.py
Closed 2 months ago
Activation quantization support
Updated 2 months ago1
mmap issue in bf16 of gpt-fast
Updated 2 months ago1
It doesn't accelerate very well at L4
Updated 2 months ago1
tokenizer.model
Updated 2 months ago1
Reasons for the poor effect of Speculative Sampling
Updated 2 months ago1
trying to convert huggingface whisper model to pytorch
Updated 2 months ago1
RuntimeError: CUDA error: named symbol not found
Updated 2 months ago3
Can't quantize to int4 and can't compile on RTX2080Ti
Closed 6 months ago3
Question about the gennerated code of `WeightOnlyInt8Linear`
Updated 3 months ago6
Size mismatch error occurs when loading models quantized by GPTQ
Updated 3 months ago2
permute function in `convert_hf_checkpoint.py`
Closed 3 months ago3
batching/dynamic batching
Updated 3 months ago2
Question about the ENABLE_INTRA_NODE_COMM for speculative decoding
Closed 4 months ago9
Support of FlashDecoding
Closed 4 months ago3
CUDA error if enabling compile_prefill for quantization model (int8)
Updated 4 months ago8
INT4 quantization not working on MI210
Closed 4 months ago2
getting different acceptance prob when using `torch.compile` after making a small change.
Updated 4 months ago
GGUF support?
Updated 5 months ago
`meta-llama/Meta-Llama-3-8B-Instruct` generates gibberish for long prompts
Closed 5 months ago5
I try to speed up with llava,but this it slower then eager mode,why?
Updated 6 months ago1
int8 Woq raise Codegen Error with `--compile_prefill`
Updated 6 months ago4
Throughput Benchmark Scripts
Closed 6 months ago2
Input token length question
Closed 6 months ago2
Naming: n_local_heads -> n_kv_heads
Updated 6 months ago
Tiny Llamas Not Found
Closed 6 months ago2
On the memory usage of `ConditionalFeedForward`
Closed 7 months ago4
Questions on Speculative Decoding in gpt-fast generate.py
Updated 7 months ago2
Question about large sequence length attention kernels
Updated 7 months ago1
AMD RX 7900 XTX Wrong outputs
Closed 8 months ago
What happens to bias during int8 quantization?
Updated 8 months ago3
index out of range: No transformer config could be loaded
Updated 8 months ago1
Reducing Latency in Application with Torch Compilation: Initialization and Inference Optimization
Updated 8 months ago
Speculative decoding with draft model:TinyLlama-1.1B
Closed 8 months ago
Can GPT-Fast support larger batch sizes
Closed 8 months ago3
pass@1 score extremely low using GPT-fast API
Closed 9 months ago2
How is llama-7b trained, what is the verification accuracy?
Closed 8 months ago2
Try Tensor Parallel on a server equipped with two V100 linked by NVLINK, but got a performance degradation
Updated 8 months ago8
Bandwidth achieved for INT8 is much smaller than FP16
Updated 8 months ago3
`eval.py` uses older version of lm_eval
Closed 9 months ago1