How to understand the volume of the manifold V, and Vol(T)?

[shutongcs]

Hi Simone,

I find it is somewhat hard to understand the terms "volume of the manifold" and "Vol(T)" in PMGA algorithm. I think in the case of 2-dimensional object, the volume of the manifold may be the area of J_1 and J_2, right? After printing L, V, D_theta_J_i, and D_t_theta_i of each iteration, I find that D_theta_J_i varies irregularly and items of D_t_theta_i varies according to the updating rho. It is interesting that V fluctuates but increases overall, and hence L increases overall (because the indicator doesn't vary widely). I'm so curious about this. How to understand V (and Vol(T)) in 2-dimensional or 3-dimensional case? And why it shows a fluctuating and increasing trend?

Look forward to your reply!

Best regards,
Shutong

[Simone Parisi]

Hi Shutong,

Thanks for the question!
The volume of the 2D front is the "length" of the curve.
You can check Munkres' book
http://fourier.math.uoc.gr/~papadim/calculus_on_manifolds/Munkres.pdf
Chapter 22
Also, if you check my paper
https://www.ias.informatik.tu-darmstadt.de/uploads/Site/EditPublication/PARISI_JAIR_MORL.pdf
Eq. 3, the loss is the integral of the indicator over the volume. In the case of a 2D front, think of it as "walking" over the curve and summing the value of the indicator I at each point.

A larger volume does not imply a larger hypervolume, as in the image below, because we do not take into account the reference point.
The blue front has smaller volume, but larger hypervolume w.r.t. to the red reference point.
So it is normal that the volume V oscillates during the learning.

Best,
Simone

[shutongcs]

Hi Simone,

Many thanks for your reply! Yes it really makes sense that in case of 2D front, V indicates the length, I and V indicate the hypervolume together.

Now I would like to use PMGA algorithm in my problem. I find that the loss (and volume) always oscillates and increases at first, but it then drops and never increases again, as shown in the following figure (the x axis is the iterations and y axis is the loss). I have no idea why it cannot keep increasing or be convergent, and which part has effect on this. Have you ever faced this situation? Maybe I need to increase the number of episodes, increase the number of agents in each iteration, or decrease the learning rate?

Best regards,
Shutong

[Simone Parisi]

Hi Shutong,

I often encounter this problem in RL, but it never happened with PMGA. However, I applied it on relatively easy problems. My best bet is that the indicator function can't evaluate solutions accurately once almost all of them are close to the frontier, and PMGA starts behaving weirdly. I noticed it with the proposed indicator (the "mixed" ones) on some MOO benchmarks, but I didn't test it extensively. The thing is that these two indicators are sensitive to the the hyperparameter lambda.
You can of course to decrease the learning rate, increase the number of steps/episodes, and also to use different (maybe richer) functions to approximate the manifold.

Best,
Simone

[shutongcs]

Hi Simone,

Thanks a million for your valuable advice! I'll try that.

Best regards,
Shutong