Testing changes to the game environments/engine
maxwellreuter opened this issue · comments
If I change, for example, bullet velocity or asset sizes in an existing game, is there a way to make sure these changes don't break any other parts of the game (especially edge cases) besides rigorously thinking up such cases and manually observing the game's behavior in "human" mode? For example: what happens if an enlarged asset goes partially outside the screen when it otherwise wouldn't as its original size? When being attacked, does its "hitbox" behave as expected?
Was some testing similar in spirit done during the creation of these games, and if so, what did that look like? If not, would you recommend any certain testing method for these types of changes?
I believe this testing is mostly done manually (for instance in human and agent mode). You can run semi-automated testing by training RL agents on it (such as https://github.com/openai/train-procgen) and checking that the training curves are similar, and watching a fully trained agent play the game.
If you are making changes to the engine that are not meant to affect the game, you can collect trajectories from random and trained agents with a fixed random seed, and make sure that they are identical for the old and new version. Since the behavior should be deterministic, the trajectories should be the same.