Issue running inference

[Daniele Grandi]

Thanks for sharing the solution to my other issue.

After adding the code you included there, I'm seeing another error:

Loading checkpoint shards: 100%|██████████| 3/3 [00:14<00:00,  4.98s/it]
Traceback (most recent call last):
  File "/mnt/c/Users/grandid/source/repos/MeLM/inference.py", line 51, in <module>
    model = PeftModel.from_pretrained(model_base, peft_model_id, subfolder=subfolder, local_dir=local_dir,
  File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/peft/peft_model.py", line 353, in from_pretrained
    model.load_adapter(model_id, adapter_name, is_trainable=is_trainable, **kwargs)
  File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/peft/peft_model.py", line 694, in load_adapter
    adapters_weights = load_peft_weights(model_id, device=torch_device, **hf_hub_download_kwargs)
  File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/peft/utils/save_and_load.py", line 328, in load_peft_weights
    adapters_weights = torch.load(filename, map_location=torch.device(device))
  File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/torch/serialization.py", line 1040, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/root/anaconda3/envs/MeLM/lib/python3.8/site-packages/torch/serialization.py", line 1258, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
ERROR conda.cli.main_run:execute(49): `conda run python /mnt/c/Users/grandid/source/repos/MeLM/inference.py` failed. (See above for error)

Process finished with exit code 1

The base model is loading fine, and I'm able to use it for inference.

[Daniele Grandi]

I found a solution.

As discussed here, the code that downloads from HF using hf_hub_download only downloads large files as pointers.

I manually downloaded the large files (adapter_model.bin and tokenizer.model), and I also changed the following line to load locally:

model = PeftModel.from_pretrained(model_base, peft_model_id, subfolder=subfolder, local_dir=local_dir)

This fixed the inference code.