rustformers / llm

Line 95 in 18b2a7d

impl Default for ConfiguredSamplers {

We have a default for ConfiguredSamplers, as well as a FromStr implementation for users to give in a sampler string. However, at the moment we do not have that default represented anywhere. It would be useful to write the default string down somewhere so users can edit and use it themselves.

An LLM-based conversion of the default leads to this (which matches the structure in the default code correctly):

repetition:penalty=1.30:last_n=64 freqpresence:last_n=64 seqrepetition topk:k=40 tailfree locallytypical topp:p=0.95 temperature:temperature=0.8 mirostat1 mirostat2

However, the default (and therefore this string) contains both mirostat samplers, but sets mirostat1 and mirostat2 to false:

llm/crates/llm-base/src/samplers.rs

Lines 167 to 169 in 18b2a7d

    
           mirostat1: false, 
        
           mirostat2: false, 
        
           incompat_mirostat: false,

So there's something going on that I'm missing, or there's something off about the current default implementation.

Alright, I finally dug into this properly. I got some debugs on the configured sampler, and was able to reconstruct our default implementation from a string.

-s "repetition:penalty=1.3:last_n=64 topk:k=40:min_keep=1 topp:p=0.95:min_keep=1 temperature:0.8"

creates the same configured sampler as our default.

I'll submit a small PR to throw this into a comment just for future users trying to learn about the code. I'll add it to the CLI help as well.

Default String for ConfiguredSamplers