This repository contains a collection of simulators for Reinforcement Learning research.

Difficulty	Name	Description	Thread-compatible?*	Optional?	Policy	Teacher?
Easy	SimpleGame	A simple 1D array-walking game.	Yes	No	Discrete	No
Easy, Medium	SimpleRace	A simple synthetic car racing game.	Yes	No	Discrete	No
Medium, Hard	Atari	Wrappers for the Arcade Learning Environment (ALE) environment. For stability, we use a fork version.	Yes	Yes	Discrete	No
Medium, Hard	XWorld2D	A 2D world for an agent to learn vision and language abilities.	No	No	Discrete Continuous	Yes
Hard	XWorld3D	A 3D world for an agent to learn vision and language abilities.	No	Yes	Discrete Continuous	Yes

(*If yes, then multithreading can be used; otherwise multiprocessing is needed.)

Architecture

XWorld features a teacher infrastructure implemented as a scheduler of multiple Finite State Machines (FSMs). The idea is that given the environment, the teacher can propose a task sampled (by some heuristics) from a task set. Each task - formulated as an FSM - has several stages, and the teacher does different things in different stages. The transition from one stage to another is determined by the envionment state, e.g., whehter the agent is idle or whether it has achieved the goal. Each stage returns several things including the next stage and the teacher's action. Currently, we define language (strings) as the teacher's sole action. However, the teacher is able to change the environment (e.g., adding/deleting objects, changing the map size, etc.) within each stage.

The above figure illustrates the architecture. The motivation is to let the users flexibly write simple Python scripts to configure the environment maps and tasks.

Currently, the teacher is only incorporated into XWorld2D and XWorld3D.

Requirements

Compiler: GCC 4.8 or above
CMake: CMake 3.0 or above
Python: Python 2.7

Dependencies

The following softwares must be installed before building XWorld.

Boost, Glog, GFlags, GTest, and Python

In Ubuntu 14.04 and 16.04, you can do

sudo apt-get install libboost-all-dev libgflags-dev libgoogle-glog-dev libgtest-dev python-dev

Build

First get this git repository

git clone https://github.com/PaddlePaddle/XWorld

Suppose the directory is xworld_path, then do

cd <xworld_path>
mkdir -p build
cd build
cmake [<optional parameters>] ..

For example,

cd ~/XWorld; mkdir build; cd build
cmake ..

Finally, in the build directory do

make
make test

By default, XWorld only builds the first three games: SimpleGame, SimpleRace, and XWorld2D.

Optionally, you can install Atari by:

cmake -DWITH_ATARI=ON ..

which will automatically download and build Atari.

You can also install XWorld3D by:

cmake -DWITH_XWORLD3D=ON ..

Usage

Python interface

We provide a set of simple Python APIs for interacting with the simulators. After building XWorld, you need to export the path of the python module:

export PYTHONPATH=<xworld_path>/python:$PYTHONPATH

You can add the above line to ~/.bashrc to avoid doing the export in the future.

To get started, several examples of the simulator Python APIs can be found in

<xworld_path>/python/examples

C++ interface

Alternatively, several C++ examples (run the .sh scripts inside) can be found in

<xworld_path>/examples

These examples use the individual class constructors to create games. However, we also provide a unified simulator interface for creating games in a more convenient way (like in Python). A demo of the unified C++ simulator interface for multi-process simulation can be found in

<xworld_path>/examples/demo_interface.cpp

Generally, C++ APIs are more flexible but expose more details compared to the Python APIs.

Flags of a game

Option flags are passed into a game via different ways for the two interfaces:

Python: you need to provide the flags when creating the game as the arguments.
C++: you need to set the flags via GFlags in the command line.

For descriptions of the flags of a game, please take a look at the README file under the game directory.

Citations

If you use XWorld2D for research, consider citing

Haichao Zhang, Haonan Yu, Wei Xu, Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game, ACL 2018.
Haonan Yu, Haichao Zhang, Wei Xu, Interactive Grounded Language Acquisition and Generalization in a 2D World, ICLR 2018.
Haichao Zhang, Haonan Yu, Wei Xu, Listen, Interact and Talk: Learning to Speak via Interaction, NIPS workshop on Visually-Grounded Interaction and Language, 2017.

If you use XWorld3D for research, consider citing

Haonan Yu, Xiaochen Lian, Haichao Zhang, Wei Xu, Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents, CoRL 2018.

If you use our wrappers of the third-party simulators, please follow their original guide for citation.

License

This repository has the Apache2.0 license, except that the third-party simulator ALE has its own license.

PaddlePaddle / XWorld