jy-yuan / KIVI

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

https://arxiv.org/abs/2402.02750

jy-yuan/KIVI Issues

Multi GPUs
Updated a month ago
Unable to Reproduce Results for LongBench
Updated a month ago1
How to understand the code: triton_quantize_and_pack_along_last_dim(value_states_full[:, :, :1, :].contiguous(), self.group_size, self.v_bits)
Closed a month ago3
How can the code support 1bit quantization.
Closed a month ago2
ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)
Updated 3 months ago
Why the model inference slowly when Mistral-7B-Instruct-v0.2 apply the kivi?
Closed 3 months ago7
Difference between gemv_forward_cuda and gemv_forward_cuda_outer_rim?
Closed 3 months ago2
NameError: name 'flash_attn_func' is not defined
Closed 4 months ago1
Where is the falcon_kivi?
Closed 4 months ago4
The difference in batch size leads to different results in LongBench testing
Updated 4 months ago5
run example.py with llama2-7B-hf only save 500MB kv cache memory conpared to base transformers ?
Updated 4 months ago2
CUDA version
Closed 5 months ago4
which commit of lm-eval-harness the lmeval branch is based on?
Closed 5 months ago3
Which file I need to run to obtain the result in Figure 4？
Closed 5 months ago2
An error occurred while using "evaluate. load (" act_match ")"
Closed 5 months ago1
Integrate KIVI into inference frameworks?
Closed 5 months ago1
W/ or w/o Weight quantization?
Closed 5 months ago4
Provide an accuracy testing interface?
Closed 5 months ago1
not support evaluation with ROCM
Updated 5 months ago1
Spport for ChatGLM3
Updated 5 months ago1
Discrepancy in Reproduced Results for LLaMA2 on "qmsum" and "qasper" tasks.
Closed 6 months ago2
Can this be used with any autogressive model?
Closed 6 months ago1
LlamaConfig.attention_dropout does not exist in transformers==4.35.2
Closed 6 months ago1
Could you please open-source the code for the calculation and visualization of the statistic information of KV Cache?
Closed 7 months ago3