predibase / lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Home Page:https://loraexchange.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

can't run lorax with docker.

cheney369 opened this issue · comments

System Info

I run the docker like this, and contioner start fail, the error: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

My cpu device is RTX3080ti.

Information

  • Docker
  • The CLI directly

Tasks

  • An officially supported command
  • My own modifications

Reproduction

my run_container.sh file
`#!/bin/bash
model=/home/ray30/ai/data/llama3-awq-int4

docker run -u root --privileged --gpus all -itd --name lorax -h lorax --net=host -v /home/ray30/ai/data:/data ghcr.io/predibase/lorax:main --model-id $model `

Expected behavior

work well

docker problems, not related to the lorax.