What system prompt do you recommend?
th789 opened this issue · comments
Hello Meditron team,
Happy New Year! Hope you are doing well. Thank you so much for releasing Meditron! I have the following questions on the recommended system prompt and how to input it to the model.
-
It seems that during evaluation, different system prompts are used for different datasets (https://github.com/epfLLM/meditron/blob/main/evaluation/inference.py#L121). In general, when using the meditron models posted on huggingface (https://huggingface.co/epfl-llm/meditron-7b and https://huggingface.co/epfl-llm/meditron-70b), what system prompt do you recommend?
-
Given a system prompt, is the following the proper way to input it to the model?
import torch
from transformers import pipeline
model_path = "epfl-llm/meditron-7b"
prompt = "What are the symptoms of diabetes?"
prompt_template = f"system prompt beginning... {prompt} ... system prompt end"
#load model and tokenizer
model = transformers.AutoModelForCausalLM.from_pretrained(
model_path,
torch_dtype=torch.float16,
device_map="auto",
)
tokenizer = transformers.AutoTokenizer.from_pretrained(model_path)
#tokenize input
prompt_tokens = tokenizer(prompt_template, return_tensors='pt')["input_ids"].to(device)
#generate output
output = model.generate(
inputs=prompt_tokens,
temperature=0.1,
do_sample=True,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
max_new_tokens=512
)
print(tokenizer.decode(output[0]))
Thank you very much!