Logistic regression very slow.

[Amando]

The RustQuant::ml::regression::logistic implementation seems to work, however it is very slow.

You can profile the example in examples/ using Flamegraph via:

sudo -E cargo flamegraph --release --freq 5000 --example logistic_regression

This shows that matrix inversion is taking about 97% of the time.

[s3bru]

I'm playing around with that at the moment. It's not really necessary to know the inverse matrix explicitely, just solve the linear equation. The hessian should be symmetric and positive definite so for example conjugate gradients algorithm should do. Also the multiplication of a vector with a diagonal matrix can be done as componentwise multiplication with the diag entries.

Do you plan to import algorithms from something like argmin or have everything in the crate?

[Amando]

I was going to import argmin at some point anyway for the planned calibration module, so its not a problem to use it already, I just liked the relatively easy "closed-form" iterative setup.

I'm also going to do the MLE version of the logistic regression when I get a chance, using the RustQuant::autodiff module.

[Colin Roberts]

Can someone point me to a resource for what is actually being computed here? There's definitely some standard tricks to get around matrix inversion. E.g., Cholesky factorization.

[Amando]

I was originally using: https://books.google.at/books/about/Machine_Learning.html?id=psuMEAAAQBAJ&source=kp_book_description&redir_esc=y

But my version was slow and unstable, @s3bru worked on it subsequently and pushed a nicely working version. I am unsure of how optimised it is, and there is room for improvements in terms of additional metrics we could return etc.

[s3bru]

The equations can also be found in wiki . Currently the linear equation is solved by nalgebras's LU decomposition, so the inverse isn't computed explicitely. The matrix to be inverted is of the form X^TDX with D diagonal and 0 > D_{ii}>1. So in principle it should be positive definite, so cholesky factorization would work and speed up that step by a factor of 2. I'm not sure if that's stable with floating point math though (The diagonal entries are calculated as 1-sigmoid(x) and for some input data the sigmoid can saturate).

I think the current problem size in the unit test is unecessarily large for automated testing. I think had 100000 training samples x 30 features only to check if this can be used for more realistic problem sizes...
If anyone can figure out a better way to do multiplications of a matrix with a diag-matrix in nalgebra that would be nice.