sglang doesn't work with vllm versions above 0.3.3

[owu]

vllm.model_executor.input_metadata is gone in higher versions of vllm. Below is me trying to run with vllm-0.4.0.post1 installed.

(build) owu@gpu:/mnt/resource_nvme$ python -m  sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --port 30000
/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/transformers/models/llava/configuration_llava.py:104: FutureWarning: The `vocab_size` argument is deprecated and will be removed in v4.42, since it can be inferred from the `text_config`. Passing this argument has no effect
  warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/transformers/models/llava/configuration_llava.py:144: FutureWarning: The `vocab_size` attribute is deprecated and will be removed in v4.42, Please use `text_config.vocab_size` instead.
  warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Process Process-1:
router init state: Traceback (most recent call last):
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/managers/router/manager.py", line 68, in start_router_process
    model_client = ModelRpcClient(server_args, port_args)
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 619, in __init__
    self.model_server.exposed_init_model(0, server_args, port_args)
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 70, in exposed_init_model
    self.model_runner = ModelRunner(
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 271, in __init__
    self.load_model()
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 280, in load_model
    model_class = get_model_cls_by_arch_name(architectures)
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 40, in get_model_cls_by_arch_name
    model_arch_name_to_cls = import_model_classes()
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 33, in import_model_classes
    module = importlib.import_module(f"sglang.srt.models.{module_path.stem}")
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/mnt/resource_nvme/miniconda3/envs/build/lib/python3.10/site-packages/sglang/srt/models/gemma.py", line 12, in <module>
    from vllm.model_executor.input_metadata import InputMetadata
ModuleNotFoundError: No module named 'vllm.model_executor.input_metadata'

detoken init state: init ok

[owu]

Looks like this was fixed in b089063

[Gintas Z.]

Looks like this was fixed in b089063

I'm still getting the same error when installing sglang[all]==0.1.14 in fresh environment

[owu]

You must build sglang from source. That version doesn't have that commit