Overflow warnings

[Joachim van der Herten]

During calls to optimize(), sometimes UserWarnings pop up:

/home/javdrher/.virtualenvs/gpflowopt/lib/python3.5/site-packages/GPflow/transforms.py:129: RuntimeWarning: overflow encountered in exp
result = np.log(1. + np.exp(x)) + self._lower

Typically this is quite harmless, if its really causing troubles its usually followed by a cholesky decomposition exception. However, those warnings mess up output. Specifically in documentation notebooks. I was thinking of silencing the warnings, any reason not to?

[James Hensman]

That particular overflow is harmless. I'd rather not silence warnings though: any ideas for how we might implement a more robust softplus?

…

Sent from my iPhone

On 25 May 2017, at 19:32, Joachim van der Herten ***@***.***> wrote: During calls to optimize(), sometimes UserWarnings pop up: /home/javdrher/.virtualenvs/gpflowopt/lib/python3.5/site-packages/GPflow/transforms.py:129: RuntimeWarning: overflow encountered in exp result = np.log(1. + np.exp(x)) + self._lower Typically this is quite harmless, if its really causing troubles its usually followed by a cholesky decomposition exception. However, those warnings mess up output. Specifically in documentation notebooks. I was thinking of silencing the warnings, any reason not to? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Reinier Maat]

How about clipping x to some threshold? Where threshold = log(MAX_FLOAT) - 1

[Joachim van der Herten]

Some improvement can be obtained with the log sum exp trick:
https://hips.seas.harvard.edu/blog/2013/01/09/computing-log-sum-exp/

Gave it a try, seems to solve forward very well, however I haven't solved the backward yet (now gives division by zeros)

[Joachim van der Herten]

Pre:

Post:

Seems to successfully suppress all warnings for the GPflowOpt tests for 2.7 and 3.5. Also tried the GPflow tests and they pass. @jameshensman interested in a PR for GPflow?

[James Hensman]

I think the correct solution is to replace

    def forward(self, x):
        result = np.log(1. + np.exp(x)) + self._lower
        # do not transform large numbers, they overflow and the mapping is exactly identity.
        return np.where(x > 35, x + self._lower, result)

with

    def forward(self, x):
        return = np.logaddexp(x, 0) + self._lower

Which does the "subtraction of max_float" for us.

[Joachim van der Herten]

Correct thats how I fixed forward, however that doesn't solve the backward transform in case y = self._lower which leads to a division by zero.

def backward(self, y):
        ys = np.maximum(y-self._lower, np.finfo(np_float_type).eps)
        return ys + np.log(1 - np.exp(-ys))

This applies the same idea used to implement np.logaddexp safely, but it also avoids np.log(0)

[James Hensman]

Good spot. How about this?

def backward(self, y):
        ys = y - self._lower
        return ys + np.log(-np.expm1(-ys))

[Joachim van der Herten]

Yikes I should read the numpy docs better. Cool, that passes the GPflow tests, however for GPflowOpt I get 30 errors:

InvalidArgumentError (see above for traceback): Input matrix is not invertible.
[[Node: MatrixTriangularSolve = MatrixTriangularSolve[T=DT_DOUBLE, adjoint=false, lower=true, _device="/job:localhost/replica:0/task:0/cpu:0"](Cholesky, unnamed.models.name.kern.K/mul)]]

Most output in the GPflowOpt tests have no output noise, hence the likelihood variance hits 1e-6 which might explain why I run into this corner case so much. Changing it into

def backward(self, y):
        ys = np.maximum(y-self._lower, np.finfo(np_float_type).eps)
        return ys + np.log(-np.expm1(-ys))

solves everything again. Between the cholesky errors this warning pops up:

/home/javdrher/.virtualenvs/gpflowopt/lib/python3.5/site-packages/GPflow/transforms.py:152: RuntimeWarning: invalid value encountered in log
return ys + np.log(-np.expm1(-ys))

I'm guessing np.expm1 is better for values near zero, however it seems to do something odd when actually encountering zero itself.

[James Hensman]

okay, great. PR this and I will merge it.