Variants of Anderson Acceleration
pkofod opened this issue · comments
They propose a variant that really has nothing to do with smooth vs non-smooth: both variants could apply to smooth or non-smooth problems.
I just glanced at it but it looks like it might be similar to what @anriseth implemented for Optim (which itself is similar to other anderson variants; there's only a finite number of non-stupid ways to combine the iterates...) There are a number of schemes that have been proposed for the solution of g(x)=x when g is a smooth black box, but nobody has really compared them against each other, both practically and theoretically. Essentially there are three dimensions along which people play with: choice of coefficients (minimizing residuals, ensuring consistency of jacobian, etc.), history management (truncation, restarts...) and robustness (trust region, regularization...) Somebody needs to sit down and compare all the variants out there, by looking at what they reduce to in the case of linear systems, and testing them out on specific nonlinear problems (of course the behavior of all these algorithms depend strongly on the problem, but that doesn't mean we can't get useful information on some well-chosen problems). I don't have the time to do it myself, but if somebody is interested I can help out. It's not hard, the algorithms are pretty straightforward if you don't care about performance. This is a subject that has really exploded over the past years, so that kind of work can have a pretty big impact on a number of communities.
Sorry, I just looked at the header tbh. :)
Cross-reference: https://github.com/cvxgrp/nonexp_global_aa1
I put together an implementation but it will need more work to integrate into the NLsolve API.
https://gist.github.com/jiahao/8e1fc43be4a3ba9c511684a8fba1b125