Overflow warnings
javdrher opened this issue · comments
During calls to optimize()
, sometimes UserWarnings pop up:
/home/javdrher/.virtualenvs/gpflowopt/lib/python3.5/site-packages/GPflow/transforms.py:129: RuntimeWarning: overflow encountered in exp
result = np.log(1. + np.exp(x)) + self._lower
Typically this is quite harmless, if its really causing troubles its usually followed by a cholesky decomposition exception. However, those warnings mess up output. Specifically in documentation notebooks. I was thinking of silencing the warnings, any reason not to?
How about clipping x to some threshold? Where threshold = log(MAX_FLOAT) - 1
Some improvement can be obtained with the log sum exp trick:
https://hips.seas.harvard.edu/blog/2013/01/09/computing-log-sum-exp/
Gave it a try, seems to solve forward very well, however I haven't solved the backward yet (now gives division by zeros)
Seems to successfully suppress all warnings for the GPflowOpt tests for 2.7 and 3.5. Also tried the GPflow tests and they pass. @jameshensman interested in a PR for GPflow?
I think the correct solution is to replace
def forward(self, x):
result = np.log(1. + np.exp(x)) + self._lower
# do not transform large numbers, they overflow and the mapping is exactly identity.
return np.where(x > 35, x + self._lower, result)
with
def forward(self, x):
return = np.logaddexp(x, 0) + self._lower
Which does the "subtraction of max_float" for us.
Correct thats how I fixed forward, however that doesn't solve the backward transform in case y = self._lower
which leads to a division by zero.
def backward(self, y):
ys = np.maximum(y-self._lower, np.finfo(np_float_type).eps)
return ys + np.log(1 - np.exp(-ys))
This applies the same idea used to implement np.logaddexp safely, but it also avoids np.log(0)
Good spot. How about this?
def backward(self, y):
ys = y - self._lower
return ys + np.log(-np.expm1(-ys))
Yikes I should read the numpy docs better. Cool, that passes the GPflow tests, however for GPflowOpt I get 30 errors:
InvalidArgumentError (see above for traceback): Input matrix is not invertible.
[[Node: MatrixTriangularSolve = MatrixTriangularSolve[T=DT_DOUBLE, adjoint=false, lower=true, _device="/job:localhost/replica:0/task:0/cpu:0"](Cholesky, unnamed.models.name.kern.K/mul)]]
Most output in the GPflowOpt tests have no output noise, hence the likelihood variance hits 1e-6 which might explain why I run into this corner case so much. Changing it into
def backward(self, y):
ys = np.maximum(y-self._lower, np.finfo(np_float_type).eps)
return ys + np.log(-np.expm1(-ys))
solves everything again. Between the cholesky errors this warning pops up:
/home/javdrher/.virtualenvs/gpflowopt/lib/python3.5/site-packages/GPflow/transforms.py:152: RuntimeWarning: invalid value encountered in log
return ys + np.log(-np.expm1(-ys))
I'm guessing np.expm1 is better for values near zero, however it seems to do something odd when actually encountering zero itself.
okay, great. PR this and I will merge it.