Default String for ConfiguredSamplers
JuliaMerz opened this issue · comments
llm/crates/llm-base/src/samplers.rs
Line 95 in 18b2a7d
We have a default for ConfiguredSamplers, as well as a FromStr implementation for users to give in a sampler string. However, at the moment we do not have that default represented anywhere. It would be useful to write the default string down somewhere so users can edit and use it themselves.
An LLM-based conversion of the default leads to this (which matches the structure in the default code correctly):
repetition:penalty=1.30:last_n=64 freqpresence:last_n=64 seqrepetition topk:k=40 tailfree locallytypical topp:p=0.95 temperature:temperature=0.8 mirostat1 mirostat2
However, the default (and therefore this string) contains both mirostat samplers, but sets mirostat1
and mirostat2
to false:
llm/crates/llm-base/src/samplers.rs
Lines 167 to 169 in 18b2a7d
So there's something going on that I'm missing, or there's something off about the current default implementation.
Alright, I finally dug into this properly. I got some debugs on the configured sampler, and was able to reconstruct our default implementation from a string.
-s "repetition:penalty=1.3:last_n=64 topk:k=40:min_keep=1 topp:p=0.95:min_keep=1 temperature:0.8"
creates the same configured sampler as our default.
I'll submit a small PR to throw this into a comment just for future users trying to learn about the code. I'll add it to the CLI help as well.