int4

There are 0 repository under int4 topic.

intel / neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime
auto-tuning awq fp4 gptq int4 int8 knowledge-distillation large-language-models low-precision mxformat post-training-quantization pruning quantization quantization-aware-training smoothquant sparsegpt sparsity
Language:Python 2013
tpoisonooo / how-to-optimize-gemm
row-major matmul optimization
gemm-optimization armv7 arm64 cuda cuda-kernel ptx vulkan int4
Language:C++ 545
intel / neural-speed
An innovative library for efficient LLM inference via low-bit quantization
cpu fp4 fp8 gaudi2 gpu int1 int2 int3 int4 int5 int6 int7 int8 llamacpp llm-fine-tuning llm-inference low-bit mxformat nf4 sparsity
Language:C++ 251
intel / auto-round
SOTA Weight-only Quantization Algorithm for LLMs. This is official implementation of "Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs"
awq gptq int4 neural-compressor quantization rounding weight-only
Language:Python 78
Danaozhong / rust-bitwriter
Rust library to write integer types of any bit length into a byte buffer.
bitbuffer bytebuffer int15 int4 rust
Language:Rust 2

intel / neural-compressor