Analysis MO-Hopper reward vector
Kallinteris-Andreas opened this issue · comments
This analysis is theoretical and backed up by tests
The multi-objective hopper's reward vector "Contains three elements"
Height (instead of
I assume 3. Was done as a proxy to
It was bugged https://github.com/Farama-Foundation/Gymnasium/issues/526, otherwise the hopper could not learn to balance
This has been fixed in version 5 And perhaps it will work with
This is important because it will indicate how more complex environments should be designed, Like the ant and humanoid.(who have a healthy reward)
Actually, we add r_healthy to all objectives, see line 57 https://github.com/Farama-Foundation/MO-Gymnasium/blob/main/mo_gymnasium/envs/mujoco/hopper.py#L57
But I see we have the exact same bug in the original environments, the reward is always given in the last time step, even if the agent is unhealthy.
We will release our v5 of the environments as soon as Gymnasium v1.0 is out. Thanks!
Do you want to keep the torso's height as a reward element?
Do you want to keep the torso's height as a reward element?
Yes, the idea is that then you can have a range of policies that trade-off between jumping forward (x-axis) vs. jumping higher (z-axis).
Is your goal
1)to to learn a policy that maximizes Gymnasium/hopper return.
2) Or to learn a policy that maximizes a different return.
Because if it is the first The current reward vector does not make sense.
When the weight assigned to the third reward component is greater than zero, it is indeed a different return. The gymnasium's return is recovered when the weight assigned to the third objective is zero. Our goal in MORL is to learn policies for any linear combination of these three rewards.
With #92 you explain the mapping,
so I am closing this issue