intel / neural-speed

An innovative library for efficient LLM inference via low-bit quantization

https://github.com/intel/neural-speed

intel/neural-speed Issues

Bestla Kernels understanding and benchmarking
Updated a month ago7
developer_document.md need elaboration on determining buffer sizes?
Updated a month ago1
Whats the different with IPEX-LLM?
Updated 2 months ago
Performance on Xeon Scalable
Updated 2 months ago1
Add support for phi3-vision
Updated 2 months ago1
is it supported with Batch size >1 ?
Updated 2 months ago2
Loading checkpoint shards takes too long
Updated 2 months ago2
Is tensor parallelism supported by neural speed?
Closed 2 months ago2
AssertionError: Fail to convert pytorch model
Updated 2 months ago3
Distributing tensors across NUMA nodes
Updated 2 months ago3
Feature request: JSON mode output
Updated 2 months ago1
heap-buffer-overflow while packing weight
Closed 2 months ago2
Performance Gap between Neural Speed Matmul Operator and Llama.cpp Operator
Updated 2 months ago13
Modifying the models hyperparameters
Updated 2 months ago8
Qwen2 GPTQ break in cpp_model.Model.np_bestla_qpack
Closed 2 months ago1
Error: Unable to install.
Closed 2 months ago5
source build from release tar file?
Updated 2 months ago1
Add support for phi-3-mini-128k model
Closed 2 months ago4
Sycl support ?
Closed 2 months ago1
context size of the model keeps fall to default of 512
Closed 2 months ago4
Garbled characters with beam search
Updated 3 months ago16
Linking back to Neural Chat / intel-extension-for-transformers
Closed 3 months ago2
i wish for simpler way to run the model
Updated 3 months ago4
i saw how beautiful this repo is, in terms of parallelism / numa stuff etc.
Updated 3 months ago1
Issue in whisper inference from pre-converted gguf
Closed 3 months ago2
Question about Thread pool and GEMV
Closed 3 months ago4
Low text quality when inferencing a gguf model with Neural Speed vs llama.cpp
Closed 3 months ago5
Huge performance difference in "Transformer-like" usage and "llama.cpp-like" usage
Closed 3 months ago2
Running Q4_K_M gguf models: unrecognized tensor type 12
Updated 4 months ago1
baseline example not working
Closed 4 months ago1
Neural Speed compilation failing in ORT
Updated 4 months ago3
What is the difference between ITREX and Neural Speed?
Closed 4 months ago3
[Feature request] Add nllb support
Closed 6 months ago4
Can't Load Qwen after support qwen2
Closed 5 months ago2
ModuleNotFoundError: No module named 'neural_speed.mistral_cpp'
Closed 5 months ago7
Error loading model when use qwen gguf model
Closed 6 months ago7
Error at Colab Inference of neural-chat-7b-v3-1 Model
Closed 5 months ago3
Error during - pip install . -
Closed 6 months ago2
Documentation for whisper inference
Closed 6 months ago2
error running inference
Closed 6 months ago3
Can't inference Llama2 through GGUF
Closed 6 months ago2
is qwen been supported?
Closed 6 months ago1
Build failure when building the executable
Closed 6 months ago3
AVX_VNNI Numeric Bug?
Closed 7 months ago1