How can I run mmlu with local model?
Geaming-CHN opened this issue · comments
I got a model finetune from Llama-3-8B, but I add the special_tokens in the new model "Llama-3-8B", the path which stores the model is data/Llama-3-8B_ft, how can i run mmlu on this local model?
What info should I add in the config (model_deployments.yaml, model_metadata.yaml, tokenizer_configs.yaml...)? If you can provide a
tutorial about this that would be so nice~
If you host the model weights on Huggingface, you can use this drafted (work in progress) documentation to get an idea of what you can do:
For your use case you would use the existing tokenizers we have for Llama 3 8b
Here's a complete example. In your case, if you have added new tokens to the tokenizer, you need to configure a new tokenizer rather than using the existing Llama 3 tokenizer, as shown below.
prod_env/model_deployments.yaml:
model_deployments:
- name: huggingface/your-model-name
model_name: your-org-name/your-model-name
tokenizer_name: your-org-name/your-model-name
max_sequence_length: 8192
client_spec:
class_name: "helm.clients.huggingface_client.HuggingFaceClient"
args:
pretrained_model_name_or_path: /path/to/your/Llama-3-8B_ft
prod_env/model_deployments.yaml:
tokenizer_configs:
- name: your-org-name/your-model-name
tokenizer_spec:
class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
args:
pretrained_model_name_or_path: /path/to/your/Llama-3-8B_ft
prefix_token: "<|begin_of_text|>"
end_of_text_token: "<|end_of_text|>"
prod_env/model_metadata.yaml:
models:
- name: your-org-name/your-model-name
display_name: Your Model Name
description: Your Model Description
creator_organization_name: Your Org Name
access: open
release_date: 2024-01-01
tags: [TEXT_MODEL_TAG, PARTIAL_FUNCTIONALITY_TEXT_MODEL_TAG]
We should probably do an end-to-end tutorial for Hugging Face at some point.