Definition of "compositional" in "compositional optimizers"
sgbaird opened this issue · comments
At one point, I may have understood this, but I find myself often wondering and asking again: what does "compositional" in "compositional optimizer" refer to? Part of my confusion probably stems from my materials informatics background, where composition often refers to the chemical make-up (i.e. chemical formula) of a particular compound. In other cases, it just implies that the contribution of individual components sums to one.
Hi! "Compositional" in "compositional optimizer" refers to the nested form of the objective function.
Consider the following optimisation problem: max_x E[f(E[g(x)])] where E denotes expected value.
A "simple" optimiser (e.g. Adam, Adagrad...) would deal with that problem by considering f(E[g(x)]) := h(x) and solve: max_x E[h(x)], while a "compositional" optimiser (e.g. CAdam) will exploit the nested form explicitly (working with f and g) to perform the optimisation.
Compositional optimiser --> exploits compositional form of the objective function.
@AntGro wonderful! Thanks so much for clarifying this for me