Efficient Anderson
pkofod opened this issue · comments
I asked about column dropping/adding in JuliaLang/julia#2929 and was made aware of https://github.com/mpf/QRupdate.jl - this seems like it would be sufficient for our use-case. I mean adding columns until truncation and then updating columns after that. Could be worth benchmarking.
Looks good, thanks! The code looks possibly not optimal in terms of allocations, temporaries, etc, but should be sufficient for our purposes. Also from a cursory look at the code it seems it's not directly QR updating, but rather (implicitly) updating the Cholesky factorization of the Gram matrix. Doesn't that square the conditioning? cc @mpf
In any case the tricky part is to restructure the code to take advantage of qr updating : once that's done we can switch implementations easily. I'll take a look at it when I have the chance (possibly next week).
In any case the tricky part is to restructure the code to take advantage of qr updating : once that's done we can switch implementations easily. I'll take a look at it when I have the chance (possibly next week).
Pinged to get your input, but it would be great if you'd want to have a stab at it! I don't remember the Anderson code being to far from a form where this should be possible, but I might underestimate how much refactoring is needed.
Inefficiencies can be worked out in the package, I'm sure @mpf would welcome improvements!
Oh if you (or someone else) want to do it go ahead, by all means! Indeed it should not be hard.
@antoine-levitt QRupdate.jl is updating a Q-less QR factorization of A. Then R satisfies
A = QR (if we had Q) and A'A = R'R
As you suggest, we need to be careful about conditioning for least-squares solves min ||Ax-b||^2_2. There are a couple of ways of solving this:
- Normal equations: solve
A'Ax=A'b, which does lead to squaring the condition number. - Using QR: solve
Rx=Q'b, which doesn't square the conditioning, but requires Q. - Semi-normal equations (SNEs): solve
R'Rx = A'bplus iterative refinement, which does satisfy a notion of stability (see the reference to Bjork in the code).
Internally, QRupdate uses the semi-normal equations with iterative refinement in order to perform the updates. Probably QRupdate should provide functionality for solving LS problems using the SNEs.
@pkofod: I would be delighted to learn where the code could be improved!
Just putting this here for future reference, this describes well how to implement AA efficiently : https://users.wpi.edu/~walker/Papers/anderson_accn_algs_imps.pdf. The QR updates can be done by hand, it's not that hard. I think it's preferable to the "Q-less" approach above, because it doesn't square the conditioning, which is a concern here.
Maybe this has some ideas? https://stanford.edu/~boyd/papers/pdf/scs_2.0_v_global.pdf
We're actually using QR updates now. I have a version in a separate package, and @devmotion implemented something very similar that is now part of this package. So this issue is actually "fixed". The paper you link could be interesting to look at, but I guess the purpose there is to solve non-smooth fixed point problems, not about efficiency of the updates?