My loss graph seems weird. (PPO training)
sooftware opened this issue · comments
My dataset format
- Prompt
Blah, blah, blah<|sep|>A:Blah, blah, blah.<|sep|>B:
- Answer
Blah, blah, blah.
Is it because the Reward average is negative?
If the reward is not normalized than rewards can (and often will) be negative. Can you share your hyperparameters/setup?
Yeah, negative average reward is nothing to worry about as long as it increases, and policy loss can have a negative value, since it's not bound by zero. If the training doesn't achieve the result you expect, firstly check if your newly trained (did you use custom dataset for the reward model as well?) reward model behaves correctly (or you can plug any other reward model just for testing) and secondly, you can tune hyperparameters to maximize the reward you get, using these instructions https://github.com/CarperAI/trlx#use-ray-tune-to-launch-hyperparameter-sweep
A normal loss graph gradually drops to a curve, is this PPO loss graph normal? Or is that graph weird?
Yes, your loss graph looks completely normal, for the reference this how other PPO loss graphs look like in regular PPO implementation on Atari
taken from https://github.com/vwxyzjn/cleanrl
https://wandb.ai/openrlbenchmark/baselines/runs/1vldj6yx
Thank you!