quantization

There are 40 repositories under quantization topic.

LLaMA-Factory
hiyouga / LLaMA-Factory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
fine-tuning llama llm peft transformers rlhf qlora quantization qwen instruction-tuning gpt lora large-language-models agent ai moe llama3 deepseek gemma nlp
Language:Python 58291
Chinese-LLaMA-Alpaca
ymcui / Chinese-LLaMA-Alpaca
中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)
llm plm pre-trained-language-models alpaca llama nlp quantization large-language-models lora alpaca-2 llama-2
Language:Python 18920
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
deep-learning inference quantization speech-recognition speech-to-text transformer whisper openai
Language:Python 18120
Qbot
UFund-Me / Qbot
[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant
funds machine-learning pytrade quantitative-finance quantitative-trading quantization strategies trademarks quant-trade quant-trader bitcoin blockchain deep-learning fintech qlib trade-bot backtest
Language:Jupyter Notebook 13905
bitsandbytes
bitsandbytes-foundation / bitsandbytes
Accessible large language models via k-bit quantization for PyTorch.
llm machine-learning pytorch qlora quantization
Language:Python 7582
kornelski / pngquant
Lossy PNG compressor — pngquant command based on libimagequant library
c conversion image-optimization palette png png-compression pngquant quality quantization smaller stdin
Language:C 5441
AutoGPTQ / AutoGPTQ
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
deep-learning inference large-language-models llms nlp pytorch quantization transformer transformers
Language:Python 4805
IntelLabs / distiller
Neural Network Distiller by Intel AI Lab: a Python package for neural network compression research. https://intellabs.github.io/distiller
pytorch pruning quantization pruning-structures jupyter-notebook network-compression deep-neural-networks regularization group-lasso distillation early-exit truncated-svd onnx automl-for-compression
Language:Jupyter Notebook 4399
OpenNMT / CTranslate2
Fast inference engine for Transformer models
avx avx2 cpp cuda deep-learning deep-neural-networks gemm inference intrinsics machine-translation mkl neon neural-machine-translation onednn openmp opennmt parallel-computing quantization thrust transformer-models
Language:C++ 4017
deepsparse
neuralmagic / deepsparse
Sparsity-aware deep learning inference runtime for CPUs
machinelearning onnx inference computer-vision object-detection pruning quantization pretrained-models nlp cpus sparsification llm-inference performance deepsparse
Language:Python 3158
huawei-noah / Pretrained-Language-Model
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
knowledge-distillation model-compression quantization pretrained-models large-scale-distributed
Language:Python 3136
huggingface / optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
onnx pytorch inference training intel graphcore onnxruntime transformers quantization habana optimization tflite
Language:Python 3084
nlp-architect
IntelLabs / nlp-architect
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks
bert deep-learning deeplearning dynet nlp nlu pytorch quantization tensorflow transformers
Language:Python 2939
aaron-xichen / pytorch-playground
Base pretrained models and datasets in pytorch (MNIST, SVHN, CIFAR10, CIFAR100, STL10, AlexNet, VGG16, VGG19, ResNet, Inception, SqueezeNet)
pytorch pytorch-tutorial pytorch-tutorials quantization
Language:Python 2690
stochasticai / xTuring
Build, customize and control you own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6
deep-learning fine-tuning gpt-2 gpt-j llama llm lora language-model alpaca finetuning adapter gen-ai generative-ai mistral mixed-precision peft quantization
Language:Python 2658
intel / neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime
low-precision pruning sparsity auto-tuning knowledge-distillation quantization quantization-aware-training post-training-quantization smoothquant large-language-models awq fp4 gptq int4 int8 mxformat sparsegpt
Language:Python 2490
quic / aimet
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
quantization deep-learning compression open-source machine-learning pruning auto-ml network-compression deep-neural-networks network-quantization opensource
Language:Python 2450
pytorch / ao
PyTorch native quantization and sparsity for training and inference
brrr dtypes inference mx pytorch quantization sparsity training float8 transformer offloading optimizer cuda llama
Language:Python 2355
dvmazur / mixtral-offloading
Run Mixtral-8x7B models in Colab or consumer desktops
colab-notebook deep-learning google-colab language-model llm mixture-of-experts offloading pytorch quantization
Language:Python 2302
666DZY666 / micronet
micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape
quantization pruning dorefa twn bnn xnor-net pytorch model-compression group-convolution network-slimming neuromorphic-computing convolutional-networks network-in-network integer-arithmetic-only quantization-aware-training post-training-quantization tensorrt onnx tensorrt-int8-python batch-normalization-fuse
Language:Python 2255
Efficient-ML / Awesome-Model-Quantization
A list of papers, docs, codes about model quantization. This repo is aimed to provide the info for model quantization research, we are continuously improving the project. Welcome to PR the works (papers, repositories) that are missed by the repo.
deep-learning quantization awesome model-compression binarized-neural-networks binary-network efficient-deep-learning lightweight-neural-network model-acceleration model-quantization
2214
intel / intel-extension-for-pytorch
A Python package for extending the official PyTorch that can easily obtain performance on Intel platform
deep-learning intel machine-learning neural-network pytorch quantization
Language:Python 1822
OpenPPL / ppq
PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.
neural-network deep-learning quantization pytorch caffe onnx cuda open-source
Language:Python 1747
open-mmlab / mmrazor
OpenMMLab Model Compression Toolbox and Benchmark.
nas pruning knowledge-distillation spos darts autoslim detection segmentation classification pytorch quantization
Language:Python 1628
PaddlePaddle / PaddleSlim
PaddleSlim is an open-source library for deep model compression and architecture search.
pruning quantization nas bert compression detection distillation ernie segmentation sparsity tensorrt transformer yolov6 yolov5 yolov7
Language:Python 1604
tensorflow / model-optimization
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
tensorflow machine-learning deep-learning optimization quantized-neural-networks quantized-networks quantized-training keras model-compression compression ml pruning sparsity quantization
Language:Python 1554
RWKV / rwkv.cpp
INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
deep-learning ggml language-model llm machine-learning quantization rwkv
Language:C++ 1544
RahulSChand / gpu_poor
Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization
ggml huggingface llm quantization gpu language-model pytorch llama llama2 llamacpp
Language:JavaScript 1354
huawei-noah / Efficient-Computing
Efficient computing methods developed by Huawei Noah's Ark Lab
knowledge-distillation model-compression binary-neural-networks pruning quantization self-supervised
Language:Jupyter Notebook 1293
Xilinx / brevitas
Brevitas: neural network quantization in PyTorch
brevitas deep-learning fpga hardware-acceleration neural-networks ptq pytorch qat quantization xilinx
Language:Python 1290
thu-ml / SageAttention
Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.
attention cuda efficient-attention inference-acceleration llm mlsys quantization triton video-generation
Language:Cuda 1279
mit-han-lab / nunchaku
[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
comfyui diffusion-models flux genai iclr iclr2025 lora mlsys quantization
Language:Cuda 1257
vllm-project / llm-compressor
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
compression quantization sparsity
Language:Python 1207
open-edge-platform / training_extensions
Train, Evaluate, Optimize, Deploy Computer Vision Models via OpenVINO™
openvino computer-vision deep-learning pytorch neural-networks-compression quantization hyper-parameter-optimization image-classification image-segmentation object-detection self-supervised-learning semi-supervised-learning transfer-learning action-recognition anomaly-detection datumaro automl machine-learning incremental-learning geti
Language:Python 1203
openvinotoolkit / nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
quantization pruning sparsity quantization-aware-training mixed-precision-training compression semantic-segmentation object-detection classification nlp bert transformers pytorch tensorflow onnx openvino deep-learning genai llm
Language:Python 1077
adithya-s-k / AI-Engineering.academy
Mastering Applied AI, One Concept at a Time
fine-tuning finetuning finetuning-llms inference large-language-models llm python quantization
Language:Jupyter Notebook 959