Can't load 4bit models on Rocm
Infection321 opened this issue · comments
Whenever I try to load 4bit models I recieve this message. I'm using the latest version of code and can load normal models just fine. I'm using a 6600xt.
``
DEVICE ID | LAYERS | DEVICE NAME
0 | 32 | AMD Radeon RX 6600 XT
N/A | 0 | (Disk cache)
N/A | 0 | (CPU)
INFO | modeling.inference_models.hf_torch_4bit:_get_model:372 - Using GPTQ file: /home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors/4bit.safetensors, 4-bit model, type llama, version 1, groupsize -1
ERROR | main:g:615 - An error has been caught in function 'g', process 'MainProcess' (47821), thread 'MainThread' (140211034212160):
Traceback (most recent call last):
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/site-packages/eventlet/green/thread.py", line 43, in __thread_body
func(*args, **kwargs)
│ │ └ {}
│ └ ()
└ <bound method Thread._bootstrap of <Thread(Thread-30, started daemon 140203333914496)>>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
│ └ <function start_new_thread..wrap_bootstrap_inner at 0x7f83a1e5a790>
└ <Thread(Thread-30, started daemon 140203333914496)>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/site-packages/eventlet/green/thread.py", line 64, in wrap_bootstrap_inner
bootstrap_inner()
└ <bound method Thread._bootstrap_inner of <Thread(Thread-30, started daemon 140203333914496)>>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
│ └ <function Thread.run at 0x7f856a182a60>
└ <Thread(Thread-30, started daemon 140203333914496)>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
│ │ │ │ │ └ {}
│ │ │ │ └ <Thread(Thread-30, started daemon 140203333914496)>
│ │ │ └ (<socketio.server.Server object at 0x7f83a3cc0280>, 'Sr6GSnjVaPX_RH0qAAAD', 'GyLyF9O7Lgzw26FwAAAC', ['load_model', {'model': ...
│ │ └ <Thread(Thread-30, started daemon 140203333914496)>
│ └ <bound method Server._handle_event_internal of <socketio.server.Server object at 0x7f83a3cc0280>>
└ <Thread(Thread-30, started daemon 140203333914496)>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/site-packages/socketio/server.py", line 731, in _handle_event_internal
r = server._trigger_event(data[0], namespace, sid, *data[1:])
│ │ │ │ │ └ ['load_model', {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': ...
│ │ │ │ └ 'Sr6GSnjVaPX_RH0qAAAD'
│ │ │ └ '/'
│ │ └ ['load_model', {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': ...
│ └ <function Server._trigger_event at 0x7f840f0c54c0>
└ <socketio.server.Server object at 0x7f83a3cc0280>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/site-packages/socketio/server.py", line 756, in _trigger_event
return self.handlers[namespace]event
│ │ │ │ └ ('Sr6GSnjVaPX_RH0qAAAD', {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', '...
│ │ │ └ 'load_model'
│ │ └ '/'
│ └ {'/': {'get_model_info': <function get_model_info at 0x7f83a2a9e430>, 'OAI_Key_Update': <function get_oai_models at 0x7f83a2a...
└ <socketio.server.Server object at 0x7f83a3cc0280>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/site-packages/flask_socketio/init.py", line 282, in _handler
return self._handle_event(handler, message, namespace, sid,
│ │ │ │ │ └ 'Sr6GSnjVaPX_RH0qAAAD'
│ │ │ │ └ '/'
│ │ │ └ 'load_model'
│ │ └ <function UI_2_load_model at 0x7f83a270c670>
│ └ <function SocketIO._handle_event at 0x7f83a4f458b0>
└ <flask_socketio.SocketIO object at 0x7f83a3cc02b0>
File "/home/bartosz/KoboldAI/runtime/envs/koboldai-rocm/lib/python3.8/site-packages/flask_socketio/init.py", line 828, in _handle_event
ret = handler(*args)
│ └ ({'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': True, 'key': '...
└ <function UI_2_load_model at 0x7f83a270c670>
File "aiserver.py", line 615, in g
return f(*a, **k)
│ │ └ {}
│ └ ({'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': True, 'key': '...
└ <function UI_2_load_model at 0x7f83a270c3a0>
File "aiserver.py", line 6493, in UI_2_load_model
load_model(use_gpu=data['use_gpu'], gpu_layers=data['gpu_layers'], disk_layers=data['disk_layers'], online_model=data['online_model'], url=koboldai_vars.colaburl, use_8_bit=data['use_8_bit'])
│ │ │ │ │ │ └ {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': True, 'key': ''...
│ │ │ │ │ └ <koboldai_settings.koboldai_vars object at 0x7f83a33fe8e0>
│ │ │ │ └ {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': True, 'key': ''...
│ │ │ └ {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': True, 'key': ''...
│ │ └ {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': True, 'key': ''...
│ └ {'model': 'NeoCustom', 'path': '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors', 'use_gpu': True, 'key': ''...
└ <function load_model at 0x7f83a2a9e8b0>
File "aiserver.py", line 1980, in load_model
model.load(
│ └ <function InferenceModel.load at 0x7f83a2b40f70>
└ <modeling.inference_models.hf_torch_4bit.HFTorch4BitInferenceModel object at 0x7f83a1f15bb0>
File "/home/bartosz/KoboldAI/modeling/inference_model.py", line 177, in load
self._load(save_model=save_model, initial_load=initial_load)
│ │ │ └ False
│ │ └ True
│ └ <function HFTorch4BitInferenceModel._load at 0x7f83a2b68940>
└ <modeling.inference_models.hf_torch_4bit.HFTorch4BitInferenceModel object at 0x7f83a1f15bb0>
File "/home/bartosz/KoboldAI/modeling/inference_models/hf_torch_4bit.py", line 198, in _load
self.model = self._get_model(self.get_local_model_path(), tf_kwargs)
│ │ │ │ │ │ └ {}
│ │ │ │ │ └ <function HFInferenceModel.get_local_model_path at 0x7f83a2b638b0>
│ │ │ │ └ <modeling.inference_models.hf_torch_4bit.HFTorch4BitInferenceModel object at 0x7f83a1f15bb0>
│ │ │ └ <function HFTorch4BitInferenceModel._get_model at 0x7f83a2b689d0>
│ │ └ <modeling.inference_models.hf_torch_4bit.HFTorch4BitInferenceModel object at 0x7f83a1f15bb0>
│ └ None
└ <modeling.inference_models.hf_torch_4bit.HFTorch4BitInferenceModel object at 0x7f83a1f15bb0>
File "/home/bartosz/KoboldAI/modeling/inference_models/hf_torch_4bit.py", line 378, in _get_model
model = load_quant_offload(llama_load_quant, utils.koboldai_vars.custmodpth, path_4bit, utils.koboldai_vars.gptq_bits, groupsize, self.gpu_layers_list, force_bias=v2_bias)
│ │ │ │ │ │ │ │ │ │ └ False
│ │ │ │ │ │ │ │ │ └ [32]
│ │ │ │ │ │ │ │ └ <modeling.inference_models.hf_torch_4bit.HFTorch4BitInferenceModel object at 0x7f83a1f15bb0>
│ │ │ │ │ │ │ └ -1
│ │ │ │ │ │ └ <koboldai_settings.koboldai_vars object at 0x7f83a33fe8e0>
│ │ │ │ │ └ <module 'utils' from '/home/bartosz/KoboldAI/utils.py'>
│ │ │ │ └ '/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors/4bit.safetensors'
│ │ │ └ <koboldai_settings.koboldai_vars object at 0x7f83a33fe8e0>
│ │ └ <module 'utils' from '/home/bartosz/KoboldAI/utils.py'>
│ └ <function load_quant at 0x7f83a2f339d0>
└ <function load_quant_offload at 0x7f83a2e5ea60>
TypeError: load_quant_offload() got an unexpected keyword argument 'force_bias'
``
Here is the debug dump
{
"errors": [],
"aiserver errors": [
{
"level": "INFO",
"message": "Running on Repo: https://github.com/0cc4m/KoboldAI Branch: latestgptq",
"record": {
"elapsed": {
"repr": "0:00:04.154708",
"seconds": 4.154708
},
"exception": null,
"extra": {},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "general_startup",
"level": {
"icon": "ℹ️",
"name": "INFO",
"no": 20
},
"line": 1312,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.314973+01:00",
"timestamp": 1687462632.314973
}
}
},
{
"level": "INIT",
"message": "Flask",
"record": {
"elapsed": {
"repr": "0:00:04.162250",
"seconds": 4.16225
},
"exception": null,
"extra": {
"status": "Starting"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "run",
"level": {
"icon": " ",
"name": "INIT",
"no": 31
},
"line": 11016,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.322515+01:00",
"timestamp": 1687462632.322515
}
}
},
{
"level": "INIT_OK",
"message": "Flask",
"record": {
"elapsed": {
"repr": "0:00:04.169604",
"seconds": 4.169604
},
"exception": null,
"extra": {
"status": "OK"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "run",
"level": {
"icon": " ",
"name": "INIT_OK",
"no": 31
},
"line": 11020,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.329869+01:00",
"timestamp": 1687462632.329869
}
}
},
{
"level": "INIT",
"message": "Webserver",
"record": {
"elapsed": {
"repr": "0:00:04.169748",
"seconds": 4.169748
},
"exception": null,
"extra": {
"status": "Starting"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "run",
"level": {
"icon": " ",
"name": "INIT",
"no": 31
},
"line": 11021,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.330013+01:00",
"timestamp": 1687462632.330013
}
}
},
{
"level": "INIT",
"message": "LUA bridge",
"record": {
"elapsed": {
"repr": "0:00:04.241993",
"seconds": 4.241993
},
"exception": null,
"extra": {
"status": "Starting"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "lua_startup",
"level": {
"icon": " ",
"name": "INIT",
"no": 31
},
"line": 2194,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.402258+01:00",
"timestamp": 1687462632.402258
}
}
},
{
"level": "INIT_OK",
"message": "LUA bridge",
"record": {
"elapsed": {
"repr": "0:00:04.246065",
"seconds": 4.246065
},
"exception": null,
"extra": {
"status": "OK"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "lua_startup",
"level": {
"icon": " ",
"name": "INIT_OK",
"no": 31
},
"line": 2221,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.406330+01:00",
"timestamp": 1687462632.40633
}
}
},
{
"level": "INIT",
"message": "LUA Scripts",
"record": {
"elapsed": {
"repr": "0:00:04.246266",
"seconds": 4.246266
},
"exception": null,
"extra": {
"status": "Starting"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "load_lua_scripts",
"level": {
"icon": " ",
"name": "INIT",
"no": 31
},
"line": 2245,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.406531+01:00",
"timestamp": 1687462632.406531
}
}
},
{
"level": "INIT_OK",
"message": "LUA Scripts",
"record": {
"elapsed": {
"repr": "0:00:04.312628",
"seconds": 4.312628
},
"exception": null,
"extra": {
"status": "OK"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "load_lua_scripts",
"level": {
"icon": " ",
"name": "INIT_OK",
"no": 31
},
"line": 2282,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:12.472893+01:00",
"timestamp": 1687462632.472893
}
}
},
{
"level": "INIT_OK",
"message": "Webserver",
"record": {
"elapsed": {
"repr": "0:00:05.804729",
"seconds": 5.804729
},
"exception": null,
"extra": {
"status": "OK"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "run",
"level": {
"icon": " ",
"name": "INIT_OK",
"no": 31
},
"line": 11101,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:13.964994+01:00",
"timestamp": 1687462633.964994
}
}
},
{
"level": "MESSAGE",
"message": "Webserver started! You may now connect with a browser at http://127.0.0.1:5000",
"record": {
"elapsed": {
"repr": "0:00:05.805081",
"seconds": 5.805081
},
"exception": null,
"extra": {},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "run",
"level": {
"icon": " ",
"name": "MESSAGE",
"no": 61
},
"line": 11102,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:13.965346+01:00",
"timestamp": 1687462633.965346
}
}
},
{
"level": "INFO",
"message": "Client connected! UI_2",
"record": {
"elapsed": {
"repr": "0:00:06.012871",
"seconds": 6.012871
},
"exception": null,
"extra": {},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "do_connect",
"level": {
"icon": "ℹ️",
"name": "INFO",
"no": 20
},
"line": 2805,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:14.173136+01:00",
"timestamp": 1687462634.173136
}
}
},
{
"level": "INFO",
"message": "Client connected! UI_1",
"record": {
"elapsed": {
"repr": "0:00:06.243572",
"seconds": 6.243572
},
"exception": null,
"extra": {},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "do_connect",
"level": {
"icon": "ℹ️",
"name": "INFO",
"no": 20
},
"line": 2805,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:14.403837+01:00",
"timestamp": 1687462634.403837
}
}
},
{
"level": "INFO",
"message": "Client connected! UI_2",
"record": {
"elapsed": {
"repr": "0:00:06.458732",
"seconds": 6.458732
},
"exception": null,
"extra": {},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "do_connect",
"level": {
"icon": "ℹ️",
"name": "INFO",
"no": 20
},
"line": 2805,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:14.618997+01:00",
"timestamp": 1687462634.618997
}
}
},
{
"level": "INFO",
"message": "Selected: NeoCustom, /home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors",
"record": {
"elapsed": {
"repr": "0:00:11.463837",
"seconds": 11.463837
},
"exception": null,
"extra": {},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "get_model_info",
"level": {
"icon": "ℹ️",
"name": "INFO",
"no": 20
},
"line": 1513,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:19.624102+01:00",
"timestamp": 1687462639.624102
}
}
},
{
"level": "INIT",
"message": "GPU support",
"record": {
"elapsed": {
"repr": "0:00:14.426968",
"seconds": 14.426968
},
"exception": null,
"extra": {
"status": "Searching"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "load_model",
"level": {
"icon": " ",
"name": "INIT",
"no": 31
},
"line": 1871,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:22.587233+01:00",
"timestamp": 1687462642.587233
}
}
},
{
"level": "INIT_OK",
"message": "GPU support",
"record": {
"elapsed": {
"repr": "0:00:14.427286",
"seconds": 14.427286
},
"exception": null,
"extra": {
"status": "Found"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "load_model",
"level": {
"icon": " ",
"name": "INIT_OK",
"no": 31
},
"line": 1884,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:22.587551+01:00",
"timestamp": 1687462642.587551
}
}
},
{
"level": "INIT",
"message": "Transformers",
"record": {
"elapsed": {
"repr": "0:00:14.431140",
"seconds": 14.43114
},
"exception": null,
"extra": {
"status": "Starting"
},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "load_model",
"level": {
"icon": " ",
"name": "INIT",
"no": 31
},
"line": 1952,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:22.591405+01:00",
"timestamp": 1687462642.591405
}
}
},
{
"level": "INIT_OK",
"message": "Final device configuration:",
"record": {
"elapsed": {
"repr": "0:00:14.442391",
"seconds": 14.442391
},
"exception": null,
"extra": {
"status": "Info"
},
"file": {
"name": "hf_torch.py",
"path": "/home/bartosz/KoboldAI/modeling/inference_models/hf_torch.py"
},
"function": "breakmodel_device_config",
"level": {
"icon": " ",
"name": "INIT_OK",
"no": 31
},
"line": 956,
"module": "hf_torch",
"name": "modeling.inference_models.hf_torch",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:22.602656+01:00",
"timestamp": 1687462642.602656
}
}
},
{
"level": "INFO",
"message": "Using GPTQ file: /home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors/4bit.safetensors, 4-bit model, type llama, version 1, groupsize -1",
"record": {
"elapsed": {
"repr": "0:00:14.513045",
"seconds": 14.513045
},
"exception": null,
"extra": {},
"file": {
"name": "hf_torch_4bit.py",
"path": "/home/bartosz/KoboldAI/modeling/inference_models/hf_torch_4bit.py"
},
"function": "_get_model",
"level": {
"icon": "ℹ️",
"name": "INFO",
"no": 20
},
"line": 372,
"module": "hf_torch_4bit",
"name": "modeling.inference_models.hf_torch_4bit",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:22.673310+01:00",
"timestamp": 1687462642.67331
}
}
},
{
"level": "ERROR",
"message": "An error has been caught in function 'g', process 'MainProcess' (55006), thread 'MainThread' (140673546893120):",
"record": {
"elapsed": {
"repr": "0:00:14.513297",
"seconds": 14.513297
},
"exception": {
"traceback": true,
"type": "TypeError",
"value": "load_quant_offload() got an unexpected keyword argument 'force_bias'"
},
"extra": {},
"file": {
"name": "aiserver.py",
"path": "aiserver.py"
},
"function": "g",
"level": {
"icon": "❌",
"name": "ERROR",
"no": 40
},
"line": 615,
"module": "aiserver",
"name": "__main__",
"process": {
"id": 55006,
"name": "MainProcess"
},
"thread": {
"id": 140673546893120,
"name": "MainThread"
},
"time": {
"repr": "2023-06-22 20:37:22.673562+01:00",
"timestamp": 1687462642.673562
}
}
}
],
"currentVars": {
"model_settings": {
"alt_multi_gen": false,
"bit_8_available": null,
"configname": "Pygmalion-7b-4bit-GPTQ-Safetensors",
"enable_whitelist": false,
"file_version": 2,
"genamt": 80,
"ikgen": 200,
"ikmax": 3000,
"max_length": 1024,
"numseqs": 1,
"online_model": "",
"rep_pen": 1.1,
"rep_pen_range": 1024,
"rep_pen_slope": 0.7,
"sampler_order": [
6,
0,
1,
2,
3,
4,
5
],
"selected_preset": "",
"temp": 0.5,
"tfs": 1,
"top_a": 0,
"top_k": 0,
"top_p": 0.9,
"typical": 1,
"use_alt_rep_pen": false,
"apikey": "[Redacted value]",
"colaburl": "[Redacted value]",
"oaiapikey": "[Redacted value]"
},
"story_settings": {
"actionmode": 0,
"actions": {
"action_count": -1,
"actions": {}
},
"actions_metadata": {},
"adventure": false,
"an_attn_bias": 1,
"andepth": 3,
"authornote": "[Redacted string with length 0]",
"authornotetemplate": "[Author's note: <|>]",
"auto_memory": "",
"autosave": false,
"biases": {},
"botname": "Bot",
"chat_style": 0,
"chatmode": false,
"chatname": "[Redacted string with length 3]",
"commentary_chance": 0,
"commentary_enabled": false,
"deletewi": null,
"dynamicscan": false,
"editln": 0,
"file_version": 2,
"gamesaved": true,
"gamestarted": false,
"gen_audio": false,
"genres": [],
"genseqs": "[Redacted array with length 0]",
"gptq_bits": 4,
"gptq_file": "/home/bartosz/KoboldAI/models/Pygmalion-7b-4bit-GPTQ-Safetensors/4bit.safetensors",
"gptq_groupsize": -1,
"gptq_model": false,
"gptq_version": 1,
"last_story_load": null,
"lastact": "[Redacted string with length 0]",
"lastctx": "[Redacted string with length 0]",
"lua_deleted": "base64:gASPlC4=\n",
"lua_edited": "base64:gASPlC4=\n",
"max_authornote_length": 512,
"max_memory_fraction": 0.5,
"max_prompt_length": 512,
"memory": "[Redacted string with length 0]",
"memory_attn_bias": 1,
"mode": "play",
"notes": "[Redacted string with length 0]",
"picture": "",
"picture_prompt": "",
"privacy_mode": false,
"privacy_password": "",
"prompt": "[Redacted string with length 0]",
"prompt_picture_filename": "",
"prompt_picture_prompt": "",
"prompt_wi_highlighted_text": [
{
"WI Text": "",
"WI matches": null,
"text": ""
}
],
"recentback": false,
"recentedit": false,
"recentrng": null,
"recentrngm": null,
"revisions": [],
"setauthornotetemplate": "[Author's note: <|>]",
"stop_sequence": [],
"story_id": 1.5143634220921282e+38,
"story_name": "[Redacted string with length 8]",
"storymode": 0,
"submission": "[Redacted string with length 0]",
"substitutions": [
{
"enabled": false,
"substitution": "–",
"target": "--",
"trueTarget": "--"
},
{
"enabled": false,
"substitution": "—",
"target": "---",
"trueTarget": "---"
},
{
"enabled": false,
"substitution": "…",
"target": "...",
"trueTarget": "..."
}
],
"useprompt": false,
"wifolders_d": {},
"wifolders_l": [],
"wifolders_u": {},
"worldinfo": [
{
"comment": "[Redacted string with length 0]",
"constant": false,
"content": "[Redacted string with length 0]",
"folder": null,
"init": false,
"key": "[Redacted string with length 0]",
"keysecondary": "[Redacted string with length 0]",
"num": 0,
"selective": false,
"uid": "-1506057865"
}
],
"worldinfo_i": [],
"worldinfo_u": {
"-1506057865": {
"comment": "[Redacted string with length 0]",
"constant": false,
"content": "[Redacted string with length 0]",
"folder": null,
"init": false,
"key": "[Redacted string with length 0]",
"keysecondary": "[Redacted string with length 0]",
"num": 0,
"selective": false,
"uid": "-1506057865"
}
},
"worldinfo_v2": {
"entries": {},
"folders": {
"[Redacted string with length 4]": "[Redacted array with length 0]"
}
}
},
"system_settings": {
"abort": false,
"acregex_ai": "base64:gASVKQAAAAAAAACMAnJllIwIX2NvbXBpbGWUk5SMDVxuKiAqPigufFxuKSqUSyCGlFKULg==\n",
"acregex_ui": "base64:gASVKAAAAAAAAACMAnJllIwIX2NvbXBpbGWUk5SMDF4gKigmZ3Q7LiopJJRLKIaUUpQu\n",
"allowsp": true,
"alt_gen": false,
"api_tokenizer_id": null,
"aria2_port": 6799,
"bmsupported": true,
"breakmodel": false,
"checking": false,
"compiling": false,
"cookies": {},
"corescript": "default.lua",
"disable_input_formatting": false,
"disable_output_formatting": false,
"experimental_features": true,
"file_version": 2,
"flaskwebgui": false,
"full_determinism": false,
"gpu_device": 0,
"has_genmod": false,
"keep_img_gen_in_memory": false,
"last_userscripts": [],
"lua_running": true,
"nobreakmodel": false,
"on_colab": false,
"port": 5000,
"quiet": false,
"regex_sl": "base64:gASVMAAAAAAAAACMAnJllIwIX2NvbXBpbGWUk5SMFFxuKig/PD0uKSAqXG4oLnxcbikqlEsghpRS\nlC4=\n",
"savedir": "[Redacted string with length 30]",
"seed": 9057897422693299000,
"seed_specified": false,
"seen_messages": [
1,
2
],
"smandelete": true,
"smanrename": true,
"sp_changed": false,
"spfilename": "[Redacted string with length 0]",
"splist": [],
"spmeta": null,
"spname": "[Redacted string with length 10]",
"spselect": "",
"standalone": false,
"status_message": "",
"story_loads": {},
"theme_list": [
"Nostalgia",
"Darkness",
"Gruvbox Dark",
"Material You",
"Unicorn",
"Monochrome"
],
"userscripts": []
},
"user_settings": {
"apikey": "",
"beep_on_complete": false,
"cluster_requested_models": [],
"colaburl": "/request",
"debug": false,
"file_version": 2,
"formatoptns": {
"frmtadsnsp": true,
"frmtrmblln": false,
"frmtrmspch": false,
"frmttriminc": true,
"singleline": false
},
"frmtadsnsp": true,
"frmtrmblln": false,
"frmtrmspch": false,
"frmttriminc": true,
"horde_api_key": "0000000000",
"horde_url": "https://horde.koboldai.net",
"horde_worker_name": "My Awesome Instance",
"img_gen_api_password": "",
"img_gen_api_url": "http://127.0.0.1:7860",
"img_gen_api_username": "",
"img_gen_art_guide": "masterpiece, digital painting, <|>, dramatic lighting, highly detailed, trending",
"img_gen_cfg_scale": 7,
"img_gen_negative_prompt": "lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, artist name",
"img_gen_priority": 1,
"img_gen_steps": 30,
"nogenmod": false,
"nopromptgen": false,
"oaiapikey": "",
"oaiengines": "https://api.openai.com/v1/engines",
"oaiurl": "",
"output_streaming": false,
"remove_double_space": true,
"rngpersist": false,
"screenshot_author_name": "Anonymous",
"screenshot_show_author_name": true,
"screenshot_show_story_title": true,
"screenshot_use_boring_colors": false,
"show_budget": false,
"show_probs": false,
"singleline": false,
"ui_level": 2,
"url": "https://api.inferkit.com/v1/models/standard/generate",
"username": "[Redacted string with length 12]",
"widepth": 3,
"wigen_amount": 80,
"wigen_use_own_wi": false,
"wirmvwhtsp": false
}
}
}
Update your GPTQ python module to the latest version.
I'm getting the same error, the TypeError: load_quant_offload() got an unexpected keyword argument 'force_bias', even after successfully updating gptq.
The latest version, that would be 0.0.3 currently, correct? Or is pip getting the wrong version?
Hold on, I should be running commandline-rocm.sh before the pip command. No wonder it didn't help if I'm not even in the right environment. Unfortunately it seems it's not so straightforward, since updating gptq fails (something is giving it an absolute path to /tmp/, when it requires a relative path). But that seems like a problem with gptq's setup script, not Kobold.
Sorry, the readme had a commit for AMD that was outdated. I removed it now, pull changes to Kobold, run install-requirements and then follow the steps for AMD in the readme.