Support for Mistral-7b
ranaya-formant opened this issue · comments
Do you all support this? https://mistral.ai/news/announcing-mistral-7b/
Apologies for the late reply on this. I've tested two Mistral 7B-derived models (https://huggingface.co/TheBloke/Mistral-7B-Claude-Chat-GGUF and https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF) with #412, and all seems to work. I'll keep this issue open until that PR lands.
Is there an update on this?
Ok I sort of figured it out. For others looking into this:
- Use the GGUF branch
- Run it like this:
cargo run --release -- infer -m ~/llms/mistral-7b-instruct-v0.1.Q4_K_S.gguf -p "Write a long story" -n 5000 -r mistralai/Mistral-7B-v0.1
The important bit is to tell it to use the tokenizer at the end.
Yup - the gguf branch's embedded tokenizer support doesn't quite work, so I've disabled it for now. I'm rebuilding the library in develop to target the latest llama.cpp, but that's going to take some time.