Model Conversion Workflow in Colab

This project demonstrates how to download a model from Hugging Face, convert it to GGUF format, and upload it back to Hugging Face using a Colab notebook.

1. Setup

First, install the necessary libraries:

!pip install huggingface_hub

2. Downloading the Model

Use the huggingface_hub library to download a model snapshot:

from huggingface_hub import snapshot_download

model_id = "Telugu-LLM-Labs/Indic-gemma-2b-finetuned-sft-Navarasa-2.0"
snapshot_download(repo_id=model_id, local_dir="Indic-gemma-2b-finetuned-sft-Navarasa",
                  local_dir_use_symlinks=False, revision="main")

!ls -lash Indic-gemma-2b-finetuned-sft-Navarasa

3. Cloning llama.cpp Repository

Clone the llama.cpp repository and install its requirements to access conversion tools:

!git clone https://github.com/ggerganov/llama.cpp.git
!pip install -r llama.cpp/requirements.txt

4. Converting the Model

Convert the downloaded Hugging Face model to GGUF format using the provided conversion script:

!python llama.cpp/convert-hf-to-gguf.py -h

!python llama.cpp/convert-hf-to-gguf.py Indic-gemma-2b-finetuned-sft-Navarasa \
  --outfile Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf \
  --outtype q8_0

!ls -lash Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf

Explanation

convert-hf-to-gguf.py -h: Displays the help message for the conversion script.
convert-hf-to-gguf.py: Converts the specified Hugging Face model to GGUF format. Parameters:
--outfile: Specifies the output file name.
--outtype: Specifies the output type (e.g., q8_0).

5. Uploading to Hugging Face Hub

Upload the converted model back to the Hugging Face Hub:

from huggingface_hub import HfApi
import os

# Set the token as an environment variable (recommended for security)
os.environ["HUGGING_FACE_HUB_TOKEN"] = "your_hugging_face_hub_token"

api = HfApi()
model_id = "your-username/Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf"
api.create_repo(model_id, exist_ok=True, repo_type="model")  # Use the api object to create the repo
api.upload_file(
    path_or_fileobj="Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf",
    path_in_repo="Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf",
    repo_id=model_id,
)

Conclusion

This notebook provides a step-by-step guide to convert a Hugging Face model to GGUF format and upload it back to the Hugging Face Hub. Modify the notebook as needed for your specific models and conversion requirements.

About