allow using llama CP
depombo opened this issue · comments
would context window be a problem?
Oh, to clarify that question, though. llama.cpp lets you set the context window size, but if you set it too large it'll OOM on you. I was able to get it to work fine with our prompt sizes in my experiments so far, so I don't think we have to worry about that.
tabling for now, but will probably re-open