Vahe1994 / AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive Quantization https://arxiv.org/pdf/2401.06118.pdf

Vahe1994/AQLM Issues

Question: How to choose a dataset for quantizing with AQLM a model like Mistral 7b-Instruct v0.2
Closed 22 days ago9
Actual bitrate of models on github?
Updated 22 days ago3
Aqlm models cannot be trained in parallel.
Closed 24 days ago5
Finetuning ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8: RuntimeError: CUDA error: invalid argument
Updated 24 days ago
Query on Evaluation Support for C4 Validation
Closed a month ago5
RuntimeError: Unknown layout
Closed a month ago8
Please help quantize sayhan/Qwen1.5-72B-Chat-LLaMAfied
Closed a month ago2
unable to install
Closed a month ago7
What are the hyperparameters to get an average of 2.02 bits per layer for LLaMA2-7b?
Closed 4 months ago8
AQLM support for Cohere Command-R +
Closed a month ago3
Minor race condition in CPU 2x8 inference code
Updated a month ago
KV Cache Quantization
Updated a month ago1
How long for the quantizing a 70b model? I had ran for 2days
Closed a month ago7
33B llama quantization post-inference time
Closed a month ago3
Request: Phi-3-mini-128k-instruct Support
Closed a month ago2
DBRX Support
Updated a month ago5
Request: AQLM quantization of the new Mixtral 8x22B
Updated a month ago2
Are there any tools that can serve AQLM quantized models?
Closed a month ago4
How to fine-tune the compressed model
Closed a month ago3
How model_seqlen affects quantization quality
Closed a month ago6
Issues while attempting LLaMA-3 Quantization
Closed a month ago22
Fine-tune colab example doesn't work
Closed 2 months ago2
Quantization Time
Closed 2 months ago13
How long does it take to quantize?
Closed 2 months ago3
Global finetuning?
Closed 2 months ago4
I wanted to know what is the beauty of technology
Closed 2 months ago8
Case Study: Instruction Tuning on AQLM Models
Closed 2 months ago7
Any lightweight OpenAI API compatible HTTP server that can be used to serve AQLM models yet?
Closed 2 months ago1
optimized 1x16 and 2x8 decompression/dequantization kernels now exist
Closed 2 months ago8
Any chance to have K x 16 / 1 x 16 scheme implemented for Numba?
Closed 2 months ago3
aqlm/inference_kernels/cuda_kernel.cu compilation errors
Closed 2 months ago10
print Total number of parameters is incomplete.
Closed 2 months ago3
Reproduce perplexity
Closed 3 months ago6
RuntimeError: Error building extension 'codebook_cuda': [1/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output cuda_kernel.cuda.o.d
Closed 3 months ago2
Supported Models
Closed 3 months ago2
AttributeError: module 'torch.library' has no attribute 'impl_abstract'
Closed 3 months ago4
Feature request: Add HF-AQLM weights on a HF org
Closed 3 months ago4
Feature request: Quantize Google Gemma models
Closed 3 months ago4
ERROR: Could not find a version that satisfies the requirement aqlm[gpu]==1.0.0
Closed 3 months ago2
Why only last token used in datasets for training?
Closed 3 months ago1
Can't run the model after building with Docker, fails in ninja build.
Closed 3 months ago2
'QuantizedLinear' object has no attribute 'weight' error
Closed 3 months ago2
Integration with HF transformers
Closed 3 months ago5
Do not hard pin the `transformers` dependency version
Closed 4 months ago1
URL typo in package metadata
Closed 4 months ago1
`RuntimeError: CUDA error: invalid argument` while running
Closed 4 months ago6
“RuntimeError: Only Tensors of floating point and complex dtype can require gradients” when trying to load a model with device_map="auto" or low_cpu_mem_usage
Closed 4 months ago1
Quantization time & VRAM requirements
Closed 4 months ago3