Use `autodiff_deferred` for the inner gradient in Enzyme hessian

Question

Use `autodiff_deferred` for the inner gradient in Enzyme hessian

Vaibhavdixit02 opened this issue a month ago · comments

Vaibhav Kumar Dixit commented a month ago

I think this might reduce the number of failure cases of second-order with Enzyme. Looking at the code this change didn't seem straightforward, hence just opening an issue and not a PR :)

Guillaume Dalle · Answer 1 · Wed May 29 2024 05:53:29 GMT+0800 (China Standard Time)

Thanks! Do you have examples of second order Enzyme use? The only one I know is in the JuMP documentation

Vaibhav Kumar Dixit · Answer 2 · Wed May 29 2024 06:29:56 GMT+0800 (China Standard Time)

https://github.com/SciML/OptimizationBase.jl/blob/main/ext/OptimizationEnzymeExt.jl#L38-L145 and https://enzyme.mit.edu/julia/dev/generated/autodiff/#Forward-over-reverse should hopefully help?

Guillaume Dalle · Answer 3 · Wed May 29 2024 14:20:06 GMT+0800 (China Standard Time)

Thanks, working on it. I think I'll have to define a custom hvp function for Enzyme over Enzyme.
I would love to get your input on https://discourse.julialang.org/t/second-order-autodiff-which-combinations-should-work/114892

Guillaume Dalle · Answer 4 · Wed May 29 2024 17:12:31 GMT+0800 (China Standard Time)

Other link for Enzyme: https://jump.dev/JuMP.jl/stable/tutorials/nonlinear/operator_ad/#Hessian-3