vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

Home Page:https://docs.vllm.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Doc]: Doc for using tensorizer_uri with LLM is incorrect

GRcharles opened this issue Β· comments

πŸ“š The doc issue

This code snippiet doesn't work - tensorizer_uri doesn't exist as a parameter

llm = LLM(model="facebook/opt-125m",
load_format="tensorizer",
tensorizer_uri=path_to_opt_tensors,
num_readers=3,
vllm_tensorized=True)

tensorizer_uri=path_to_opt_tensors,

Suggest a potential alternative/fix

No response