Error 804
FieldOfRice opened this issue · comments
nvidia-smi display the data shown below.
After a "docker-compose up" and some downloading the messages with the error 804 are there(see below).
I'm running this inside a host running debian 12.
Seems a nvidia/torch related issue, any pointers how to resolve it?
Thu Feb 1 12:42:13 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06 Driver Version: 525.125.06 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | Off |
| 0% 37C P8 12W / 450W | 6MiB / 24564MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1137 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
Recreating text-generation-webui ... done Attaching to text-generation-webui text-generation-webui | === Running text-generation-webui variant: 'DEFAULT' snapshot-2023-12-31 === text-generation-webui | === (This version is 11 commits behind origin main) === text-generation-webui | === Image build date: 2024-01-04 22:56:23 === text-generation-webui | 11:39:20-264526 INFO Starting Text generation web UI text-generation-webui | 11:39:20-266083 WARNING text-generation-webui | You are potentially exposing the web UI to the entire text-generation-webui | internet without any access password. text-generation-webui | You can create one with the "--gradio-auth" flag like text-generation-webui | this: text-generation-webui | text-generation-webui | --gradio-auth username:password text-generation-webui | text-generation-webui | Make sure to replace username:password with your own. text-generation-webui | 11:39:20-266993 INFO Loading the extension "gallery" text-generation-webui | ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ text-generation-webui | │ /app/server.py:254 in │ text-generation-webui | │ │ text-generation-webui | │ 253 # Launch the web UI │ text-generation-webui | │ ❱ 254 create_interface() │ text-generation-webui | │ 255 while True: │ text-generation-webui | │ │ text-generation-webui | │ /app/server.py:133 in create_interface │ text-generation-webui | │ │ text-generation-webui | │ 132 ui_parameters.create_ui(shared.settings['preset']) # Paramete │ text-generation-webui | │ ❱ 133 ui_model_menu.create_ui() # Model tab │ text-generation-webui | │ 134 training.create_ui() # Training tab │ text-generation-webui | │ │ text-generation-webui | │ /app/modules/ui_model_menu.py:36 in create_ui │ text-generation-webui | │ │ text-generation-webui | │ 35 for i in range(torch.cuda.device_count()): │ text-generation-webui | │ ❱ 36 total_mem.append(math.floor(torch.cuda.get_device_properti │ text-generation-webui | │ 37 │ text-generation-webui | │ │ text-generation-webui | │ /venv/lib/python3.10/site-packages/torch/cuda/__init__.py:449 in │ text-generation-webui | │ get_device_properties │ text-generation-webui | │ │ text-generation-webui | │ 448 """ │ text-generation-webui | │ ❱ 449 _lazy_init() # will define _get_device_properties │ text-generation-webui | │ 450 device = _get_device_index(device, optional=True) │ text-generation-webui | │ │ text-generation-webui | │ /venv/lib/python3.10/site-packages/torch/cuda/__init__.py:298 in _lazy_init │ text-generation-webui | │ │ text-generation-webui | │ 297 os.environ["CUDA_MODULE_LOADING"] = "LAZY" │ text-generation-webui | │ ❱ 298 torch._C._cuda_init() │ text-generation-webui | │ 299 # Some of the queued calls may reentrantly call _lazy_init(); │ text-generation-webui | ╰──────────────────────────────────────────────────────────────────────────────╯ text-generation-webui | RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda text-generation-webui | functions before calling NumCudaDevices() that might have already set an error? text-generation-webui | Error 804: forward compatibility was attempted on non supported HW text-generation-webui exited with code 1
The updates should have fixed this issue. Please re-open if not.