Reinforcement learning with docker and torcs

Installation

Currently this section is updated.

NOTE: The gpu busid needs to be converted from hex to decimal, i.e. the g3 instance with a busid of 0:1e:0 becomes 0:30:0

The Torcs reinforcement learning environment is based on an Amazon EC2 g2.2xlarge instance running Ubuntu 16.04.

To setup the environment, the following commands have to be executed from a shell:

git clone https://github.com/bn2302/rl_torcs
cd rl_torc/docker/
sudo su
source root_setup.sh
reboot

After rebooting the instance please run the following commands:

cd rl_torc/docker
source user_setup.sh

The script will install the nvidia drivers, nvidia-docker and an xserver, which is used to connect to the agent via virtualgl.

Next to that the script will build the images for two docker containers:

* Torcs running in a container with virtualgl and turbovnc

* A reinforcement learning environment containing Tensorflow, a modified
  vim and other goodies

Start the docker container

The reinforcement learning docker environment is started using start_rl to reattach the environment the alias attach_rl can be used.

Start the training

The different agents can be trained using the scripts in the src folder called train_X.py. Please not if an agent is prematurely canceled the corresponding torcs container must be stopped using docker stop NAME . To list the running containers please use docker ps -l -a

Monitor the training

To monitor the training process, please connect to the containers, go into the logs directory and start tensorboard using

tensorboard --logdir=a3c_0:'./a3c/train_0/',a3c_1:'./a3c/train_1',a3c_2:'./a3c/train_2',a3c_3:'./a3c/train_3',a3c_4:'./a3c/train_4/',a3c_5:'./a3c/train_5',a3c_6:'./a3c/train_6',a3c_7:'./a3c/train_7',ddpg_1:'./ddpg_1',dddpg_ref:'./ddpg_ref',ddpg_2:'./ddpg_2/'

Tensorboard can be accessed via port 6006 from a browser. When connecting to an AWS instance via ssh, forward the port with -L 6006:localhost:6006. Then tensorboard can be opened in a browser using http://localhost:6006/

Start the testing

Testing is done in the Jupyter notebook test.ipynb . To start the jupyter server run the following command from the main directory

jupyter server --allow-root

Jupyter can be accessed via port 8888 from a browser. When connecting to an AWS instance via ssh, forward the port with -L 6006:localhost8888

References

https://github.com/awjuliani/DeepRL-Agents/blob/master/A3C-Doom.ipynb as the basis for the A3c implementation.

https://github.com/yanpanlau/DDPG-Keras-Torcs as the basis for the DDPG.

https://github.com/plumbee/nvidia-hw-accelerated-box as the basis for the setup scripts.

bn2302 / rl_torcs