johnsmith0031 / alpaca_lora_4bit

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

error with monkeypatch and model gpt-j and lora

ReDXeoL opened this issue · comments

Hello, I would like to discuss a problem I have with only the 4bit quantized gpt-j models (gpt-j-6B-alpaca-4bit-128g) with the help of the AutoGPTQ method when using the monkeypatch to train with lora

(this only happens with this type of models, models like vicuna or WizardLM work very well with monkeypatch)

ERROR:
Load Model ...
WARNING:The safetensors archive passed at models\bertin-gpt-j-6B-alpaca-4bit-128g\gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the save_pretrained method. Defaulting to 'pt' metadata.
Traceback (most recent call last):
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\server.py", line 932, in
shared.model, shared.tokenizer = load_model(shared.model_name)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\models.py", line 153, in load_model
model, _ = load_model_llama(model_name)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\monkey_patch_gptq_lora.py", line 24, in load_model_llama
model, tokenizer = load_llama_model_4bit_low_ram(config_path, model_path, groupsize=shared.args.groupsize, is_v1_model=False)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\repositories\alpaca_lora_4bit\autograd_4bit.py", line 204, in load_llama_model_4bit_low_ram
model = accelerate.load_checkpoint_and_dispatch(
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\big_modeling.py", line 479, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 946, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 135, in set_module_tensor_to_device
if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None:
AttributeError: 'NoneType' object has no attribute 'device'
Captura
pc: rtx 3060 (12GB) i7 10700 32 ram