Support for Mistral-7b
ranaya-formant opened this issue · comments
Do you all support this? https://mistral.ai/news/announcing-mistral-7b/
Apologies for the late reply on this. I've tested two Mistral 7B-derived models (https://huggingface.co/TheBloke/Mistral-7B-Claude-Chat-GGUF and https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF) with #412, and all seems to work. I'll keep this issue open until that PR lands.
Is there an update on this?
Ok I sort of figured it out. For others looking into this:
- Use the GGUF branch
- Run it like this:
cargo run --release -- infer -m ~/llms/mistral-7b-instruct-v0.1.Q4_K_S.gguf -p "Write a long story" -n 5000 -r mistralai/Mistral-7B-v0.1
The important bit is to tell it to use the tokenizer at the end.
Yup - the gguf
branch's embedded tokenizer support doesn't quite work, so I've disabled it for now. I'm rebuilding the library in develop
to target the latest llama.cpp
, but that's going to take some time.