This repository is for open-questions relating to RLHF and InstructGPT as pertaining to BigModelName.
- What is the preference rate of PPO vs PPO-Ptx? Why was 27.8 chosen as the mixing factor between the pre-training gradients and the PPO gradients?
- What do the gradient norms and gradient noise scales look like for PPO grads vs pre-training grads?
- How important is SFT pretraining on human-written completions?