LM Algorithm: Unbounded lambda

[aiden-huffman]

I have some optimisations which appear to get caught in a local minimum so that chi = chi_2 and as a result rho = 0. In the current setup, the algorithm doubles lambda until there is some improvement, this seems to lead to divergence and the algorithm failing for some optimisation problems. A simple fix might be to have the algorithm halt when lambda exceeds a meaningful value.

[Tae-Geun Kim]

Thank you for great suggestion! I'm with you on that.
I think there are two ways to implement your idea.

1. Add method
   : Add set_lambda method which only works for LM method & add two private fields to Optimizer - lambda, lambda_max

   impl Optimizer {
       // ...
       pub fn set_lambda(&mut self, lambda_init: f64, lambda_max: f64) -> &mut Self {
           // ...
           match self.method {
               LevenbergMarquardt => { /* ... */ },
               _ => panic!(" ... "),
           }
           // ...
       }
   }

2. Modify Enum
   : Modify LevenbergMarquardt enum to more detailed.

   pub enum OptMethod {
       // ...
       LevenbergMarquardt(f64, f64), // lambda_init, lambda_max
   }

Which do you prefer? Or if you have any other idea, then please let me know.

[aiden-huffman]

I think the first solution is the better of the two, this gives advanced users more control over the algorithm. If the user doesn't need to use the functionality, then on the back end something as simple as a default value lambda_max = f64::MAX.sqrt() or equivalent which you've handled previously with the default option hash map.

[Tae-Geun Kim]

Thank you again for good comments!
Then I'll implement this idea asap.

[Tae-Geun Kim]

Completed!
I published new version : 0.30.13.
Here is an example (just find optimal [a,b,c] for y = ax^2 + bx + c)

#[allow(non_snake_case)]
fn test_LM() {
    let x = seq(0, 10, 0.1);
    let p_true = vec![1.0, 2.0, 3.0];
    let y = x.fmap(|t| p_true[0] * t.powi(2) + p_true[1] * t + p_true[2]);

    let p_init = vec![1f64, 1f64, 1f64];
    let data = hstack!(x, y);
    let mut opt = Optimizer::new(data, f);
    let p_est = opt
        .set_init_param(p_init)
        .set_max_iter(50)
        .set_method(LevenbergMarquardt)
        .set_lambda_init(1e-3)
        .set_lambda_max(1e+3)
        .optimize();

    p_est.print();
}

fn f(x: &Vec<f64>, p: Vec<AD>) -> Option<Vec<AD>> {
    Some (
        x.iter()
            .map(|t| AD1(*t, 0f64))
            .map(|t| p[0] * t.powi(2) + p[1] * t + p[2])
            .collect()
    )
}

If lambda exceeds lambda_max then iteration is stopped and print iteration information.
In this example, you can see the message as follows.

Caution: At 31-th iter, lambda exceeds max value: 12122.73120743323