class-based-ppo This is a fork of OpenAI spinning up pytorch PPO. It should be easier to customize because you can simply subclass it.