POPGym is designed to benchmark memory in deep reinforcement learning. It contains a set of environments and a collection of memory model baselines. The full paper is available on OpenReview. Please see the documentation for advanced installation instructions and examples.
pip install popgym # base environments only, only requires numpy and gymnasium
pip install "popgym[navigation]" # also include navigation environments, which require mazelib
pip install "popgym[baselines]" # environments and memory baselinesPOPGym contains Partially Observable Markov Decision Process (POMDP) environments following the Openai Gym interface. POPGym environments have minimal dependencies and fast enough to solve on a laptop CPU in less than a day. We provide the following environments:
| Environment | Tags | Temporal Ordering | Colab FPS | Macbook Air (2020) FPS |
|---|---|---|---|---|
| Battleship (Code) | Game | None | 117,158 | 235,402 |
| Concentration (Code) | Game | Weak | 47,515 | 157,217 |
| Higher Lower (Code) | Game, Noisy | None | 24,312 | 76,903 |
| Labyrinth Escape (Code) | Navigation | Strong | 1,399 | 41,122 |
| Labyrinth Explore (Code) | Navigation | Strong | 1,374 | 30,611 |
| Minesweeper (Code) | Game | None | 8,434 | 32,003 |
| Multiarmed Bandit (Code) | Noisy | None | 48,751 | 469,325 |
| Autoencode (Code) | Diagnostic | Strong | 121,756 | 251,997 |
| Count Recall (Code) | Diagnostic, Noisy | None | 16,799 | 50,311 |
| Repeat First (Code) | Diagnostic | None | 23,895 | 155,201 |
| Repeat Previous (Code) | Diagnostic | Strong | 50,349 | 136,392 |
| Stateless Cartpole (Code) | Control | Strong | 73,622 | 218,446 |
| Noisy Stateless Cartpole (Code) | Control, Noisy | Strong | 6,269 | 66,891 |
| Stateless Pendulum (Code) | Control | Strong | 8,168 | 26,358 |
| Noisy Stateless Pendulum (Code) | Control, Noisy | Strong | 6,808 | 20,090 |
Feel free to rerun this benchmark using this colab notebook.
POPGym baselines implements recurrent and memory model in an efficient manner. POPGym baselines is implemented on top of rllib using their custom model API. We provide the following baselines:
- MLP
- Positional MLP
- Framestacking (Paper)
- Temporal Convolution Networks (Paper)
- Elman Networks (Paper)
- Long Short-Term Memory (Paper)
- Gated Recurrent Units (Paper)
- Independently Recurrent Neural Networks (Paper)
- Fast Autoregressive Transformers (Paper)
- Fast Weight Programmers (Paper)
- Legendre Memory Units (Paper)
- Diagonal State Space Models (Paper)
- Differentiable Neural Computers (Paper)
The leaderboard is available at paperswithcode.
Follow style and ensure tests pass
pip install pre-commit
pre-commit install
pytest popgym/tests@inproceedings{
morad2023popgym,
title={{POPG}ym: Benchmarking Partially Observable Reinforcement Learning},
author={Steven Morad and Ryan Kortvelesy and Matteo Bettini and Stephan Liwicki and Amanda Prorok},
booktitle={The Eleventh International Conference on Learning Representations},
year={2023},
url={https://openreview.net/forum?id=chDrutUTs0K}
}