swuuu / safe-control-gym

PyBullet CartPole and Quadrotor environments—with CasADi symbolic a priori dynamics—for learning-based control and RL

safe-control-gym

Physics-based CartPole and Quadrotor Gym environments (using PyBullet) with symbolic a priori dynamics (using CasADi) for learning-based control, and model-free and model-based reinforcement learning (RL).

These environments include (and evaluate) symbolic safety constraints and implement input, parameter, and dynamics disturbances to test the robustness and generalizability of control approaches. [PDF]

@article{brunke2021safe,
         title={Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning},
         author={Lukas Brunke and Melissa Greeff and Adam W. Hall and Zhaocong Yuan and Siqi Zhou and Jacopo Panerati and Angela P. Schoellig},
         journal = {Annual Review of Control, Robotics, and Autonomous Systems},
         year={2021},
         url = {https://arxiv.org/abs/2108.06266}}

To reproduce the results in the article, see branch ar.

@misc{yuan2021safecontrolgym,
      title={safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning},
      author={Zhaocong Yuan and Adam W. Hall and Siqi Zhou and Lukas Brunke and Melissa Greeff and Jacopo Panerati and Angela P. Schoellig},
      year={2021},
      eprint={2109.06325},
      archivePrefix={arXiv},
      primaryClass={cs.RO}}

To reproduce the results in the article, see branch submission.

Install on Ubuntu/macOS

Clone repo

git clone https://github.com/utiasDSL/safe-control-gym.git
cd safe-control-gym

Option A (recommended): using conda

Create and access a Python 3.10 environment using conda

conda create -n safe python=3.10
conda activate safe

Install the safe-control-gym repository

pip install --upgrade pip
pip install -e .

Option B: using venv and poetry

Create and access a Python 3.10 virtual environment using pyenv and venv

pyenv install 3.10
pyenv local 3.10
python3 -m venv safe
source safe/bin/activate
pip install --upgrade pip
pip install poetry
poetry install

Note

You may need to separately install gmp, a dependency of pycddlib:

conda install -c anaconda gmp

or

sudo apt-get install libgmp-dev

Architecture

Overview of safe-control-gym's API:

Configuration

Performance

We compare the sample efficiency of safe-control-gym with the original OpenAI Cartpole and PyBullet Gym's Inverted Pendulum, as well as gym-pybullet-drones. We choose the default physic simulation integration step of each project. We report performance results for open-loop, random action inputs. Note that the Bullet engine frequency reported for safe-control-gym is typically much finer grained for improved fidelity. safe-control-gym quadrotor environment is not as light-weight as gym-pybullet-drones but provides the same order of magnitude speed-up and several more safety features/symbolic models.

Environment	GUI	Control Freq.	PyBullet Freq.	Constraints & Disturbances^	Speed-Up^^
Gym cartpole	True	50Hz	N/A	No	1.16x
InvPenPyBulletEnv	False	60Hz	60Hz	No	158.29x
cartpole	True	50Hz	50Hz	No	0.85x
cartpole	False	50Hz	1000Hz	No	24.73x
cartpole	False	50Hz	1000Hz	Yes	22.39x
gym-pyb-drones	True	48Hz	240Hz	No	2.43x
gym-pyb-drones	False	50Hz	1000Hz	No	21.50x
quadrotor	True	60Hz	240Hz	No	0.74x
quadrotor	False	50Hz	1000Hz	No	9.28x
quadrotor	False	50Hz	1000Hz	Yes	7.62x

^ Whether the environment includes a default set of constraints and disturbances

^^ Speed-up = Elapsed Simulation Time / Elapsed Wall Clock Time; on a 2.30GHz Quad-Core i7-1068NG7 with 32GB 3733MHz LPDDR4X; no GPU

Getting Started

Familiarize with APIs and environments with the scripts in examples/

cd ./examples/                                                                    # Navigate to the examples folder
python3 pid/pid_experiment.py --algo pid --task quadrotor --overrides ./pid/config_overrides/quadrotor_3D/quadrotor_3D_tracking.yaml       # PID trajectory tracking with the 3D quadcopter
python3 no_controller/verbose_api.py --task cartpole --overrides no_controller/verbose_api.yaml               #  Printout of the extended safe-control-gym APIs

Systems Variables and 2D Quadrotor Lemniscate Trajectory Tracking

Verbose API Example

List of Implemented Controllers

• PID
• LQR
• iLQR
• Linear MPC
• GP-MPC
• SAC
• PPO
• DDPG
• Safety Layer
• RARL
• RAP

List of Implemented Safety Filters

• MPSC
• CBF
• Neural Network CBF

References

• Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., & Schoellig, A. P. (2022). Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5, 411-444.
• Yuan, Z., Hall, A. W., Zhou, S., Brunke, L., Greeff, M., Panerati, J., & Schoellig, A. P. (2022). safe-control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning in robotics. IEEE Robotics and Automation Letters, 7(4), 11142-11149.

Related Open-source Projects

• gym-pybullet-drones: single and multi-quadrotor environments
• stable-baselines3: PyTorch reinforcement learning algorithms
• bullet3: multi-physics simulation engine
• gym: OpenAI reinforcement learning toolkit
• casadi: symbolic framework for numeric optimization
• safety-gym: environments for safe exploration in RL
• realworldrl_suite: real-world RL challenge framework
• casadi: symbolic framework for numeric optimization

TODOs (August 2022)

• Publish to PyPI
• Create resource list with papers, projects, blog posts (Cat's, etc.) using safe-control-gym

---

University of Toronto's Dynamic Systems Lab / Vector Institute for Artificial Intelligence