bentoml / OpenLLM

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

https://bentoml.com

bentoml/OpenLLM Issues

bug: pip package version ssues
Closed 14 days ago
feat: Multimodal LLMs?
Updated 16 days ago
For AMD/GPU, how to use multi GPUS in the api_server.py
Updated 18 days ago
feat: Can you support llama3?
Updated 20 days ago1
bug: error coming up while install the vllm using pip install "openllm[vllm]"
Updated 20 days ago
bug: AttributeError in Phi-2 Model Initialization Using BentoML OpenLLM Docker Container
Updated 21 days ago4
feat: support LMDeploy backend
Updated 21 days ago1
bug: WARNING: openllm 0.4.44 does not provide the extra 'gemma'
Updated 22 days ago
bug: Not able to work with locally downloaded model
Updated 23 days ago1
bug: An exception occurred while instantiating runner 'llm-mistral-runner'
Updated a month ago1
bug: Not enough data for satisfy transfer length header
Updated a month ago
feat: support Qwen1.5
Updated a month ago1
feat: any plan to support NPU
Updated a month ago
feat: c4ai-command-r-v01 support
Updated a month ago2
Deploying LLM in On-Premises Server to Assist Users to Launch Locally in Work Laptop - Web Browser
Updated a month ago3
bug: "Failed to parse JSON from SSE message" error
Updated a month ago4
bug: Docker images with GPTQ quantized models do not have auto-gptq or optimum installed
Updated 2 months ago1
FileNotFoundError: [Errno 2] No such file or directory: b'/root/bentoml/models/pt-google--gemma-7b-it/latest'
Updated 2 months ago
Deprecation Warning for PyTorch Backend
Updated 2 months ago2
I'm having trouble getting statted with openllm, but I don't want to use conda and I have WSL2
Updated 2 months ago1
feat: add vllm 0.3.0 support
Updated 2 months ago3
feat: support volta architecture GPUs for the vLLM backend
Updated 2 months ago
bug: start chatglm-6b locally err
Updated 2 months ago
How to load a model offline
Updated 2 months ago1
bug: not generate eos_token when using qwen7B-chat
Updated 3 months ago
openllm_core.exceptions.OpenLLMException: Failed to initialise vLLMEngine due to the following error: Model architectures ['T5ForConditionalGeneration'] are not supported for now.
Closed 3 months ago1
When attempting to add the CohereForAI/aya-101 model using CohereForAI's Aya model, an error occurred during the loading process.
Updated 3 months ago
Runtime error about concurrency
Updated 3 months ago
bug: Error in sending post request for bentoml container service
Updated 3 months ago1
RunnerService: MAX_MODEL_LEN is not reflected to the llm._max_model_len
Closed 3 months ago2
bug: Requests with "use_beam_search: true" fail with an unclear exception message.
Updated 3 months ago
bug: TypeError attribute name must be string, not 'NoneType'
Updated 3 months ago3
Can't pass workers_per_resource to the bentoml container
Updated 3 months ago2
bug: Inference fails for `docker run` with `google/flan-t5-large` model with `pytorch` on CPU
Updated 3 months ago1
feat: Document usage of request_id
Updated 3 months ago
How to update the prompt template without change openllm-core config
Updated 3 months ago
bug: Can't load local models with (No such file or directory) error
Updated 3 months ago
bug: running any model results on the error: "Error: [bentoml-cli] `serve` failed: Invalid Tag _service:svc"
Updated 3 months ago
How to run a docker image on multiple GPUs?
Updated 4 months ago
How to modify the port number?
Updated 4 months ago
Unable to fetch models.
Updated 4 months ago
bug: [WARNING] [api_server:llm-llama-service:3] Timed out waiting for runner to be ready
Updated 4 months ago
bug: RuntimeError: Exception caught during generation: Response payload is not completed
Updated 4 months ago
Why change prompt_token_ids depending on encoder_decoder
Updated 4 months ago
Add a gradio demo for the python sdk section
Updated 4 months ago
bug: block GPU
Updated 4 months ago1
bug: when use backend=pt in normal or adapter mode,RuntimeError: Exception caught during generation: Failed to parse JSON from SSE message:(vllm is ok)
Updated 4 months ago2
running out of memory
Updated 4 months ago
bug: 'torch.dtype' object has no attribute 'lower'
Updated 4 months ago
Docker image for version 0.4.41 not built
Updated 4 months ago