x35f / unstable_baselines

Re-implementations of SOTA RL algorithms.

---

Unstable Baselines(USB) is designed to serve as a quick-start guide for Reinforcement Learning beginners and a codebase for agile algorithm development. The algorithms strictly follows the original implementations, and the performance of Unstable Baselines matches those in the original implementations. USB is currently maintained by researchers from lamda-rl.

---

Features

1. Novice-friendly: USB is written in simple python codes. The RL training procedures are highly decoupled, waiting to be your first RL playground.
2. Stick to the original implementations: USB is as a benchmark framework for RL, thus the re-implementations strictly follows the original implementations. Tricks to achieve a higher performance are not implemented.
3. Customized Environments: You can customized you own environment as long as it has Gym-like interfaces.

---

Implementation Details

Baseline RL	Continuous Action Space	Discrete Action Space	Image Input	Status
DQN	✕	✔	✔	Stable
VPG	✔	✕	✔	Stable
DDPG	✔	✕	✔	Stable
TD3	✔	✕	✔	Stable
TRPO	✔	✕	✔	Stable
PPO	✔	✔	✔	Stable
SAC	✔	✔	✔	Stable
REDQ	✔	✔	✔	Stable
Option Critic	-	-	-	Developing

Model Based RL	Continuous Action Space	Discrete Action Space	Image Input	Status
MBPO	✔	✕	✔	Updating

Meta RL	Continuous Action Space	Discrete Action Space	Image Input	Status
PEARL	✔	✕	✕	Updating
MAML	-	-	-	Developing

*Updating: the algorithm is being developed to adapt to the latest USB version, and will be "Stable" soon

*Developing: the algorithm is being implemented, and will appear on the project soon

---

Supported environment benchmarks

• Gym ("Classic Control" and "Box2D")
• MuJoCo
• Atari
• dm_control
• metaworld

---

Performance

MuJoCo

Quick Start

Install

git clone --recurse-submodules https://github.com/x35f/unstable_baselines.git
cd unstable_baselines
conda env create -f env.yaml 
conda activate usb
pip install -e .

To run an algorithm

In the directory of the algorithm

python3 /path/to/algorithm/main.py /path/to/algorithm/configs/some-config.py args(optional)

For example

cd unstable_baselines/baselines/sac
python3 main.py configs/Ant-v3.py --gpu 0

or for the ease of aggregating logs

python3 unstable_baselines/baselines/sac/main.py unstable_baselines/baselines/sac/configs/Ant-v3.py --gpu 0

Install environments (optional)

#install metaworld for meta_rl benchmark
cd envs/metaworld
pip install -e .

TODO List

• Add Documentation