Issue running inference
grndnl opened this issue · comments
Thanks for sharing the solution to my other issue.
After adding the code you included there, I'm seeing another error:
Loading checkpoint shards: 100%|██████████| 3/3 [00:14<00:00, 4.98s/it]
Traceback (most recent call last):
File "/mnt/c/Users/grandid/source/repos/MeLM/inference.py", line 51, in <module>
model = PeftModel.from_pretrained(model_base, peft_model_id, subfolder=subfolder, local_dir=local_dir,
File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/peft/peft_model.py", line 353, in from_pretrained
model.load_adapter(model_id, adapter_name, is_trainable=is_trainable, **kwargs)
File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/peft/peft_model.py", line 694, in load_adapter
adapters_weights = load_peft_weights(model_id, device=torch_device, **hf_hub_download_kwargs)
File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/peft/utils/save_and_load.py", line 328, in load_peft_weights
adapters_weights = torch.load(filename, map_location=torch.device(device))
File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/torch/serialization.py", line 1040, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/torch/serialization.py", line 1258, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
ERROR conda.cli.main_run:execute(49): `conda run python /mnt/c/Users/grandid/source/repos/MeLM/inference.py` failed. (See above for error)
Process finished with exit code 1
The base model is loading fine, and I'm able to use it for inference.
I found a solution.
As discussed here, the code that downloads from HF using hf_hub_download only downloads large files as pointers.
I manually downloaded the large files (adapter_model.bin and tokenizer.model), and I also changed the following line to load locally:
model = PeftModel.from_pretrained(model_base, peft_model_id, subfolder=subfolder, local_dir=local_dir)
This fixed the inference code.