githubharald / CTCWordBeamSearch

Connectionist Temporal Classification (CTC) decoder with dictionary and language model.

Home Page:https://harald-scheidl.medium.com/b051d28f3d2e

Repository from Github https://github.comgithubharald/CTCWordBeamSearchRepository from Github https://github.comgithubharald/CTCWordBeamSearch

Replace mulitiplication with addition can prevent underflow for decimals

wangershi opened this issue · comments

Python prototype, version is 3.6

Same as Substract a maxValue to prevent overflow of e(y), it is not a logical problem, just a trick to prevent underflow.

As the probability of words is decimal and the path may be long, so the probability of sequences is the mulitiplication of many decimals, for example, p(seq)=0.10.1...*0.1, and this is a unadvisable operation for underflow, I want to fix it and wonder if you have tried to do something to prevent this.

A common usage is to use log, as log(a*b) = log(a) + log(b), such codes can be found in ctc_beam_search.h:

logsumexp = Eigen::numext::log(logsumexp);
float norm_offset = max_coeff + logsumexp;

And log(a+b) can be found in ctc_loss_util.h:

// Add logarithmic probabilities using:
// ln(a + b) = ln(a) + ln(1 + exp(ln(b) - ln(a)))
// The two inputs are assumed to be log probabilities.
// (GravesTh) Eq. 7.18
inline float LogSumExp(float log_prob_1, float log_prob_2) {
  // Always have 'b' be the smaller number to avoid the exponential from
  // blowing up.
  if (log_prob_1 == kLogZero) {
    return log_prob_2;
  } else if (log_prob_2 == kLogZero) {
    return log_prob_1;
  } else {
    return (log_prob_1 > log_prob_2)
               ? log_prob_1 + log1pf(expf(log_prob_2 - log_prob_1))
               : log_prob_2 + log1pf(expf(log_prob_1 - log_prob_2));
  }
}