Line search does not check KL constraint satisfaction
zhihanyang2022 opened this issue · comments
The TRPO paper (Appendix C) claims that "we use a line search to ensure improvement of the surrogate objective and satisfaction of the KL divergence constraint". However, in the current codebase, the linesearch function only checks whether the candidate update improves the surrogate advantage, which deviates from the paper and doesn't check for KL constraint.
https://github.com/joschu/modular_rl/blob/master/modular_rl/trpo.py#L92
Could you help me understand the reasoning behind this? @joschu
Thanks!