predibase / lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

https://loraexchange.ai

predibase/lorax Issues

When caching adapters, cache the adapter ID + the API token pair
Updated 3 days ago
Add HTTP status codes to docs
Updated 3 days ago
Reject unknown fields from API requests
Updated 3 days ago
Support inference on INF2 instance
Updated 4 days ago
Improve warmup checking for max new tokens when using speculative decoding
Updated 6 days ago
Ensure api_token is not included in the response on error
Updated 6 days ago3
Bug Report: lorax-launcher failed with --source "s3" for model_id "mistralai/Mistral-7B-Instruct-v0.2"
Updated 7 days ago
[QUESTION] How to change HuggingFace model download Path in Lorax When deployed to Kubernetes through HelmChart
Closed 8 days ago1
AutoTokenzier.from_pretrains needs setting with `trust_remote_code` inside `load_module_map`
Updated 10 days ago2
Inference with AWQ quantized base model + compile enabled results in the <unk> tokens
Updated 12 days ago
Add HF authentication instructions to lorax-launcher docs
Closed 16 days ago6
Add all launcher args as optional in the Helm charts
Updated 14 days ago
Retrieve all lora models from Huggingface hub by base model setting.
Updated 16 days ago
Fallback to Flash Attention v1 for pre-Ampere GPUs
Updated 17 days ago1
Improve async load for adapters to avoid main thread lockups in server
Updated 20 days ago
Batch inference endpoint (OpenAI compatible)
Updated 23 days ago
Speculative tokens fails during warmup in some scenarios
Closed 23 days ago1
Private LORA Adapter Error - Server error: No valid adapter config file found: tried None and None
Closed 25 days ago1
Llama3-8b-Instruct won't stop generating
Updated a month ago
Idefics2 and LLaVA
Updated a month ago
Fix PyTorch CUDA version in Docker
Updated a month ago
In Structured Output, a JSON schema with a date string format will yield invalid JSON
Updated a month ago2
Improve the latency of `load_batched_adapter_weights`
Updated a month ago1
10s latency of lora inference caused by None base_model_name_or_path in adapter_config
Closed a month ago
an illegal memory access was encountered for Mixtral with 1700 tokens
Updated a month ago3
[Question] Usage about the `adapter-memory-fraction`
Closed a month ago1
Combining multiple LoRA adapters
Updated a month ago1
Can't run Mistral quantized on T4
Updated a month ago4
Error: Warmup(Generation("'bool' object has no attribute 'dtype'"))
Updated a month ago2
Support loading `.pt` weights
Updated a month ago1
Async client to backoff when model overloaded
Updated a month ago
Misleading/wrong openapi schema in REST API docs for structured output
Updated a month ago
Need some help. " You need to decrease --max-batch-prefill-tokens."
Updated a month ago1
Upgrade to AWQ kernels v0.0.6
Updated a month ago1
LoRAX server with 2 GPUs and multiple adapters becomes permanently faster in swapping ONLY after parallel execution of requests.
Updated a month ago1
log arbitrary headers
Updated a month ago1
Refactor the quantization config for weights
Closed a month ago
Wrong inference result for Llama2 AWQ model with qkv bias enabled
Closed a month ago
Add support for fp8 (H100)
Updated a month ago2
None
Closed a month ago1
Add support for LoReFT
Updated a month ago1
Supporting inference with EETQ quantized model
Updated 2 months ago
Add support for AQLM quantization
Updated 2 months ago
Fix Mixtral usage with `--compile`
Updated 2 months ago1
Add helm chart to OCI repo
Updated 2 months ago3
Fix Mixtral with Medusa
Closed 2 months ago
Fix `--compile` with Medusa
Closed 2 months ago
Allow deterministic generation for OpenAI compatible requests
Closed 2 months ago
JSON errors in `generate()` happen for certain base models (but not for others)
Closed 2 months ago1
Support constrained generation of valid Python types
Updated 2 months ago