Issue on CUDA version and Torch on vllm

[Sahussawud Khunruksa]

While i build a service with docker, the error is raised.

output

291.9   The detected CUDA version (12.1) mismatches the version that was used to compile 
291.9   PyTorch (11.7). Please make sure to use the same CUDA versions. 
291.9
291.9   ----------------------------------------
291.9   ERROR: Failed building wheel for vllm
291.9 Failed to build vllm
293.3 ERROR: Could not build wheels for vllm which use PEP 517 and cannot be installed directly

Dockerfiles

FROM nvidia/cuda:11.6.2-devel-ubuntu20.04

# Install Python 3.8
RUN apt-get update && apt-get install -y python3.8 python3-pip && apt-get clean

# Set Python 3.8 as the default
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1

# Set the working directory in the container
WORKDIR /app

# Install git
RUN apt-get update && apt-get install -y git && apt-get clean

# Copy the current directory contents into the container at /app
COPY . /app

ENV CUDA_HOME="/usr/local/cuda"
ENV FORCE_CUDA="1"

# Install the required packages
RUN pip install --no-cache-dir -r requirements.txt

# Expose port 8000 for the app to listen on
EXPOSE 8000

# Define the command to run the app
CMD ["python", "api_server.py", "--trust-remote-code", "--model", "/path/to/model/folder"]

Docker-compose

version: '3.8'  # Consider using a more recent version

services:
  llm-server:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    volumes:
      - ./Documents/WizardCoder-Python-13B-V1.0:/app/models
    environment:
      - MODEL_PATH=/app/models

[Daniel Fitzgibbon]

Hi @sahussawud!
I've had this issue as well and what solved it was upgrading PyTorch to one of the nightly builds (newer version that's still pending release).
You can install those using their link on the website and picking out which OS and version you're after but you'll probably need to install it via this command:

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121