Jeffrey28 / SMARTS

Scalable Multi-Agent RL Training School for Autonomous Driving

SMARTS

SMARTS (Scalable Multi-Agent RL Training School) is a simulation platform for reinforcement learning (RL) and multi-agent research on autonomous driving. Its focus is on realistic and diverse interactions. It is part of the XingTian suite of RL platforms from Huawei Noah's Ark Lab.

Check out the paper at SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving for background on some of the project goals.

Multi-agent experiment as simple as ...

import gym

from smarts.core.agent_interface import AgentInterface, AgentType
from smarts.core.agent import Agent
from smarts.zoo.agent_spec import AgentSpec

class SimpleAgent(Agent):
    def act(self, obs):
        return "keep_lane"

agent_spec = AgentSpec(
    interface=AgentInterface.from_type(AgentType.Laner, max_episode_steps=None),
    agent_builder=SimpleAgent,
)

agent_specs = {
    "Agent-007": agent_spec,
    "Agent-008": agent_spec,
}

env = gym.make(
    "smarts.env:hiway-v0",
    scenarios=["scenarios/sumo/loop"],
    agent_specs=agent_specs,
)

agents = {
    agent_id: agent_spec.build_agent()
    for agent_id, agent_spec in agent_specs.items()
}
observations = env.reset()

for _ in range(1000):
    agent_actions = {
        agent_id: agents[agent_id].act(agent_obs)
        for agent_id, agent_obs in observations.items()
    }
    observations, _, _, _ = env.step(agent_actions)

Contents

1. Documentation
2. Setup
   • Installation
   • Running
3. Examples
   • Usage
   • Reinforcement Learning
4. CLI Tool
   • CLI Usage
   • CLI Examples
5. Visualizing Observations
6. PyMARL and MALib
7. Containers
   • Docker
   • Singularity
8. Troubleshooting
   • General
   • SUMO
9. Bug Reports
10. Contributing
11. Citing

Documentation

Documentation is available at smarts.readthedocs.io.

Setup

Installation

git clone https://github.com/huawei-noah/SMARTS.git
cd <path/to/SMARTS>

# For Mac OS X users, ensure XQuartz is pre-installed.
# Install the system requirements. You may use the `-y` option to enable automatic assumption of "yes" to all prompts to avoid timeout from waiting for user input. 
bash utils/setup/install_deps.sh

# Setup virtual environment. Presently at least Python 3.7 and higher is officially supported.
python3.7 -m venv .venv

# Enter virtual environment to install dependencies.
source .venv/bin/activate

# Upgrade pip.
pip install --upgrade pip

# Install smarts with extras as needed. Extras include the following: 
# `camera-obs` - needed for rendering camera sensor observations, and for testing.
# `test` - needed for testing.
# `train` - needed for RL training and testing.
pip install -e '.[camera-obs,test,train]'

# Run sanity-test and verify they are passing.
# If tests fail, check './sanity_test_result.xml' for test report. 
make sanity-test

Running

Use the scl command to run SMARTS together with it's supporting processes.

To run the default example, firstly build the scenario scenarios/sumo/loop.

scl scenario build --clean scenarios/sumo/loop

Then, run a single-agent SMARTS simulation with Envision display and loop scenario.

scl run --envision examples/single_agent.py scenarios/sumo/loop 

The --envision flag runs the Envision server which displays the simulation visualization. See ./envision/README.md for more information on Envision, SMARTS's front-end visualization tool.

After executing the above command, visit http://localhost:8081/ to view the experiment.

Several example scripts are provided in examples folder, as well as a handful of scenarios in scenarios folder. You can create your own scenarios using the Scenario Studio. Below is the generic command to run and visualize one of the example scripts with a scenario.

scl run --envision <examples/path> <scenarios/path> 

Examples

Usage

Illustration of various ways to use SMARTS.

1. Single agent example.
2. Multi agent example.
3. Parallel environments to run multiple SMARTS environments in parallel.

Reinforcement Learning

1. MARL benchmark
2. Driving in traffic using world model based RL.

CLI Tool

SMARTS provides a command-line tool to interact with scenario studio and Envision.

CLI Usage

scl COMMAND SUBCOMMAND [OPTIONS] [ARGS]...

Commands:

• scenario
• envision
• zoo
• run

Subcommands of scenario:

• build: Generate a single scenario.
• build-all: Generate all scenarios under the given directories.
• clean: Clean generated artifacts.

Subcommands of envision:

• start: Start Envision server.

Subcommands of zoo:

• build: Build a policy, to submit to the agent zoo.
• install: Attempt to install the specified agents from the given paths/url.
• manager: Start the manager process which instantiates workers.

Subcommands of run:

• No subcommands. Use run directly to simulate as shown above.

CLI Examples

# Start envision and serve scenario assets out of ./scenarios
scl envision start --scenarios ./scenarios

# Build all scenarios under given directories
scl scenario build-all ./scenarios ./eval_scenarios

# Rebuild a single scenario, replacing any existing generated assets
scl scenario build --clean scenarios/sumo/loop

# Clean generated scenario artifacts
scl scenario clean scenarios/sumo/loop

Visualizing Observations

Use the Visdom integration to easily visualize the observations.

Firstly, start the Visdom server in a terminal.

visdom
# Open the printed URL in a browser.

Secondly, in a separate terminal, run SMARTS simulation. Enable Visdom in the environment by setting visdom=True. For example:

env = gym.make(
    "smarts.env:hiway-v0",
    scenarios=["scenarios/sumo/loop"],
    agent_specs=agent_specs,
    visdom=True,
)

Below is a sample visualization of an agent's camera sensor observations.

(Left) Drivable area grid map. (Center) Occupancy grid map. (Right) Top-down RGB image.

PyMARL and MALib

Run SMARTS with PyMARL.

git clone [e-mail]:ying-wen/pymarl.git

ln -s your-project/scenarios ./pymarl/scenarios

cd pymarl

# Setup virtual environment.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

python src/main.py --config=qmix --env-config=smarts

Run SMARTS with MALib.

git clone [e-mail]:ying-wen/malib.git

ln -s your-project/scenarios ./malib/scenarios

cd malib

# Setup virtual environment.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

python examples/run_smarts.py --algo SAC --scenario ./scenarios/sumo/loop --n_agents 5

Containers

Docker

SMARTS docker images are hosted at dockerhub.

$ cd </path/to/SMARTS>
$ docker run --rm -it -v $PWD:/src -p 8081:8081 huaweinoah/smarts:<version>
# E.g. docker run --rm -it -v $PWD:/src -p 8081:8081 huaweinoah/smarts:v0.5.1

# If visualization is needed, run Envision server in the background.
$ scl envision start -s ./scenarios -p 8081 &

# Build the scenario. 
# This step is required on the first time, and whenever the scenario is modified.
$ scl scenario build scenarios/sumo/loop --clean

# Run an example. 
# Add --headless if visualisation is not needed.
$ python examples/single_agent.py scenarios/sumo/loop

# Visit http://localhost:8081 in the host machine to see the running simulation in Envision.

Singularity

$ cd </path/to/SMARTS>

# Build container from definition file.
$ sudo singularity build ./utils/singularity/smarts.sif ./utils/singularity/smarts.def

# Use the container to build the required scenarios.
$ singularity shell --containall --bind ../SMARTS:/src ./utils/singularity/smarts.sif
# Inside the container
Singularity> scl scenario build /src/scenarios/sumo/loop/
Singularity> exit

# Then, run the container using one of the following methods.

# 1. Run container in interactive mode.
$ singularity shell --containall --bind ../SMARTS:/src ./utils/singularity/smarts.sif
# Inside the container
Singularity> python3.7 /src/examples/single_agent.py /src/scenarios/sumo/loop/ --headless

# 2. Run commands within the container from the host system.
$ singularity exec --containall --bind ../SMARTS:/src ./utils/singularity/smarts.sif python3.7 /src/examples/single_agent.py /src/scenarios/sumo/loop/ --headless

# 3. Run container instance in the background.
$ singularity instance start --containall --bind ../SMARTS:/src ./utils/singularity/smarts.sif smarts_train /src/examples/single_agent.py /src/scenarios/sumo/loop/ --headless

Troubleshooting

General

In most cases SMARTS debug logs are located at ~/.smarts. These can be helpful to diagnose problems.

SUMO

SUMO might encounter problems during setup. Please look through the following for support for SUMO:

• If you are having issues see: Setup and SUMO TROUBLESHOOTING
• If you wish to find binaries: SUMO Download Page
• If you wish to compile from source see: SUMO Build Instructions.
  • Please note that building SUMO may not install other vital dependencies that SUMO requires to run.
  • If you build from the git repository we recommend to use SUMO version 1.7.0 or higher

Bug Reports

Please read how to create a bug report and then open an issue here.

Contributing

Please read contributing.

Citing

If you use SMARTS in your research, please cite the paper. In BibTeX format:

@misc{zhou2020smarts,
      title={SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving},
      author={Ming Zhou and Jun Luo and Julian Villella and Yaodong Yang and David Rusu and Jiayu Miao and Weinan Zhang and Montgomery Alban and Iman Fadakar and Zheng Chen and Aurora Chongxi Huang and Ying Wen and Kimia Hassanzadeh and Daniel Graves and Dong Chen and Zhengbang Zhu and Nhat Nguyen and Mohamed Elsayed and Kun Shao and Sanjeevan Ahilan and Baokuan Zhang and Jiannan Wu and Zhengang Fu and Kasra Rezaee and Peyman Yadmellat and Mohsen Rohani and Nicolas Perez Nieves and Yihan Ni and Seyedershad Banijamali and Alexander Cowen Rivers and Zheng Tian and Daniel Palenicek and Haitham bou Ammar and Hongbo Zhang and Wulong Liu and Jianye Hao and Jun Wang},
      url={https://arxiv.org/abs/2010.09776},
      primaryClass={cs.MA},
      booktitle={Proceedings of the 4th Conference on Robot Learning (CoRL)},
      year={2020},
      month={11}
 }