Error `TypeError: not a string` while merging lora

[Naozumi]

Check before submitting issues

• Make sure to pull the latest code, as some issues and bugs have been fixed.
• Due to frequent dependency updates, please ensure you have followed the steps in our Wiki
• I have read the FAQ section AND searched for similar issues and did not find a similar problem or solution
• Third-party plugin issues - e.g., llama.cpp, text-generation-webui, LlamaChat, we recommend checking the corresponding project for solutions
• Model validity check - Be sure to check the model's SHA256.md. If the model is incorrect, we cannot guarantee its performance

Type of Issue

Model conversion and merging

Base Model

LLaMA-7B

Operating System

macOS

Describe your issue in detail

With following the instruction in here, I got the error TypeError: not a string from sentencepiece.

Dependencies (must be provided for code-related issues)

No response

Execution logs or screenshots

naozumi@Naozumis-MacBook-Pro Chinese-LLaMA-Alpaca % python3.10 /Users/naozumi/Downloads/Chinese-LLaMA-Alpaca/scripts/merge_llama_with_chinese_lora_low_mem.py \
    --base_model meta-llama/Llama-2-7b-chat-hf \
    --lora_model Naozumi/llama2-qlora-finetunined-cantonese \
    --output_type pth \
    --output_dir /Users/naozumi/Downloads/cantoneseModel
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'
Base model: meta-llama/Llama-2-7b-chat-hf
LoRA model(s) ['Naozumi/llama2-qlora-finetunined-cantonese']:
Loading Naozumi/llama2-qlora-finetunined-cantonese
Cannot find lora model on the disk. Downloading lora model from hub...
Fetching 7 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 57456.22it/s]
Traceback (most recent call last):
  File "/Users/naozumi/Downloads/Chinese-LLaMA-Alpaca/scripts/merge_llama_with_chinese_lora_low_mem.py", line 246, in <module>
    tokenizer = LlamaTokenizer.from_pretrained(lora_model_path)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1811, in from_pretrained
    return cls._from_pretrained(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1965, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 96, in __init__
    self.sp_model.Load(vocab_file)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
naozumi@Naozumis-MacBook-Pro Chinese-LLaMA-Alpaca % 

[Naozumi]

I've tried using both model name and the path of the local folder, still no luck.

[Ziqing Yang]

Can you list the files in Naozumi/llama2-qlora-finetunined-cantonese?
It looks like the problem is related to the tokenizer.

[Yiming Cui]

Cannot find lora model on the disk. Downloading lora model from hub...

This tells you that the lora model Naozumi/llama2-qlora-finetunined-cantonese cannot be found.
You should specify either a local folder or a huggingface identifier for --lora_model.
In this context, you need to check what files are there in Naozumi/llama2-qlora-finetunined-cantonese folder.

[Naozumi]

README.md adapter_config.json adapter_model.bin gitattributes.txt special_tokens_map.json tokenizer.json tokenizer_config.json

These are the files in my lora folder.

[Naozumi]

Maybe it was the problem of the Colab notebook. I was fine tuned using the Llama_2_Fine_Tuning_using_QLora-2.ipynb I could search on web. But it doesn't seemed to be saving the tokenizer so I added a line tokenizer.push_to_hub("my-awesome-model"). If possible, can I have any Colab notebook that tokenizer can be created that is recommended? Fine-tune on my pc is not possible in this moment.

[Yiming Cui]

The merging script was intended for the models in this project, and thus we did not test the compatiblity of other models.
You might need to save your tokenizer into tokenizer.model first, and try to merge your model again. Also check adapter_model.bin is with a proper file size (at least several MB).
Refer to our Chinese-LLaMA-LoRA-7B: https://huggingface.co/ziqingyang/chinese-llama-lora-7b/tree/main and check what file was missing.

[Naozumi]

I've checked the Llama_2_Fine_Tuning_using_QLora-2.ipynb once again and it seems this notebook I used takes the tokenizer from the base model. This explained why the generated file tokenizer.json didn't contain any Chinese characters.

The fact is what I'm trying to do is to fine-tune the model with my Cantonese datasets in JSON format (instruction, input and output). But I couldn't find one online notebook with capability to generate a tokenizer. What am I supposed to do?

[Yiming Cui]

If you are going to do further instruction fine-tuning on Llama-2 series, refer to https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/wiki/sft_scripts_en

[Naozumi]

I'm at the step setting value chinese_tokenizer_path in run_sft.sh. Do I have to use the tokenizer in this repo? It's not a problem tho, I'm just worrying the language branches I'm fine tuning in Cantonese.

[Ziqing Yang]

I'm at the step setting value chinese_tokenizer_path in run_sft.sh. Do I have to use the tokenizer in this repo? It's not a problem tho, I'm just worrying the language branches I'm fine tuning in Cantonese.

You must use the corresponding tokenizer released together with the model weights in order to fine-tune the model correctly.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your consideration.

[github-actions]

Closing the issue, since no updates observed. Feel free to re-open if you need any further assistance.