Replace mulitiplication with addition can prevent underflow for decimals
wangershi opened this issue · comments
Python prototype, version is 3.6
Same as Substract a maxValue to prevent overflow of e(y), it is not a logical problem, just a trick to prevent underflow.
As the probability of words is decimal and the path may be long, so the probability of sequences is the mulitiplication of many decimals, for example, p(seq)=0.10.1...*0.1, and this is a unadvisable operation for underflow, I want to fix it and wonder if you have tried to do something to prevent this.
A common usage is to use log, as log(a*b) = log(a) + log(b), such codes can be found in ctc_beam_search.h:
logsumexp = Eigen::numext::log(logsumexp);
float norm_offset = max_coeff + logsumexp;
And log(a+b) can be found in ctc_loss_util.h:
// Add logarithmic probabilities using:
// ln(a + b) = ln(a) + ln(1 + exp(ln(b) - ln(a)))
// The two inputs are assumed to be log probabilities.
// (GravesTh) Eq. 7.18
inline float LogSumExp(float log_prob_1, float log_prob_2) {
// Always have 'b' be the smaller number to avoid the exponential from
// blowing up.
if (log_prob_1 == kLogZero) {
return log_prob_2;
} else if (log_prob_2 == kLogZero) {
return log_prob_1;
} else {
return (log_prob_1 > log_prob_2)
? log_prob_1 + log1pf(expf(log_prob_2 - log_prob_1))
: log_prob_2 + log1pf(expf(log_prob_1 - log_prob_2));
}
}