Giters

Farama-Foundation / MO-Gymnasium

Multi-objective Gymnasium environments for reinforcement learning

Home Page:http://mo-gymnasium.farama.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Flexibility of using un-vectorized environment

ChrisZonghaoLi opened this issue · comments

commented

Greetings,

I believe the benefit of using vectorized environments is that the agent can be trained in parallel with a stack of independent environments so that the training speed can be improved.

However, after digging around some code here at mo-gymnasium and gymnasium, I found that, many wrappers of vectorized environments (such as SyncVectorEnv) are run in series rather than in parallel. Does this kind of losing the point of vectorizing environment in the first place?

Also, it is not always computationally inexpensive to run an environment (where the headroom is not training the agent), in this scenario I may go with un-vectorized environment. However, the current API forces user to define vectorized environment, which I think is not very user-friendly and should be an user-defined option instead.

Thanks,

Chris

commented

Hello Chris,

Yes, VecEnv can be made parallel but it is not supported in mo-gymnasium for now. We indeed rely on MOSyncVectorEnv that are run in series.

The main reason why we introduced this is that existing RL codebases such as cleanRL heavily rely on such features, for good reasons. This allows us for easier conversion of single objective RL algorithms to multi-objective (see what do in MORL-Baselines for example: MOPPO).

I am not sure I understand your point with the current API forcing user to define vectorized envs, it does not. Again, an example of algorithm not relying on VecEnvs in MORL-Baselines: EUPG.

I hope these links help you. If you have more questions, you can also join our discord: https://discord.gg/y8QanqB9.

commented

Lets discuss this over DC