ADMM: Inner L-BFGS optimizations fail to converge

[Chris White]

If I slightly modify the inner local_update solves in the current ADMM implementation as follows:

def local_update(y, X, w, z, u, rho, fprime=proximal_logistic_gradient,
                 f=proximal_logistic_loss,
                 solver=fmin_l_bfgs_b):
    w = w.ravel()
    u = u.ravel()
    z = z.ravel()
    solver_args = (X, y, z, u, rho)
    w, f, d = solver(f, w, fprime=fprime, args=solver_args, pgtol=1e-10,
                     maxiter=200,
                     maxfun=250, factr=1e-30)
    if d['warnflag']:
        raise ValueError(
            '''Internal LBFGSB algorithm failed to converge!
            Details: {}'''.format(d['task']))
    else:
        return w.reshape(2, 1)

Then on random data I see convergence issues:

from dask_glm.utils import make_y
X = da.random.random((100,2), chunks=(100/5, 2))
y = make_y(X)
logistic_regression(X, y[:,None], 0.1, 0.1, 1)

With the following error:

Traceback (most recent call last):
...
dask.async.ValueError: Internal LBFGSB algorithm failed to converge!
            Details: b'ABNORMAL_TERMINATION_IN_LNSRCH'

which refers to abnormal termination in the line search (always with the line search...).

From the fmin_l_bfgs_b output:

Line search cannot locate an adequate point after 20 function
  and gradient evaluations.  Previous x, f and g restored.
 Possible causes: 1 error in function or gradient evaluation;
                  2 rounding error dominate computation.

[Chris White]

This appears to be due to incredibly stringent keyword arguments that we were feeding, namely: pgtol=1e-10 and factr=1e-30. Using the scipy defaults makes this go away.