dmlc / gluon-nlp

Description

The top-p sampling here is not correct.

gluon-nlp/src/gluonnlp/sequence_sampler.py

Lines 720 to 724 in 52da8ab

    
           probs = mx.np.where( 
        
               probs > self._sampling_topp, 
        
               probs, 
        
               mx.np.zeros_like(probs) 
        
           )

We should sort the probabilities in descending order and then do a cumsum of the probabilities. After that, we set the threshold based on the cumsum values.

@hymzoque

	probs = mx.np.where(
	probs > self._sampling_topp,
	probs,
	mx.np.zeros_like(probs)
	)

[Bug] Bug of top-p sampling

Description