philectron / cassierl

Reinforcement learning algorithms for Oregon State University's bipedal robot, Cassie

cassierl

This repository contains reinforcement learning algorithms implemented for Cassie, a bipedal walking robot of Oregon State University.

Installing Libraries

In order to compile the files in src, you need to clone this repository which contains third-party libraries and follow the instruction to install.

How to Set Paths to Your Own Files

In src/Cassie2d/FilePath.h,

• Change the value of MUJOCO_LICENSE_PATH to where your mjkey.txt is.

• Change the value of XML_FILE_PATH to where your cassie2d_stiff.xml is.

How to Fix Import Errors when python3 cassie2d.py

If you got into import error(s) when trying to run cassie2d.py, follow these steps, depending on what modules you don't have, to fix the problem:

• No modules named rllab

cd somewhere else outside of this repository

Execute the following commands

git clone https://github.com/rll/rllab.git
cd rllab
sudo python3 setup.py install

Then, you can import rllab and run.

• No modules named theano: sudo pip3 install theano
• No modules named path: sudo pip3 install path.py
• No modules named cached_property: sudo pip3 install cached_property

After these fixes, there shouldn't be any other problems. If you still encounter import errors, try to look for what modules your python can't import, then do sudo pip3 install MODULE_NAME. This usually works, but not always. Make sure to use pip3 because we are doing python3.

How to make and execute

From the root directory cassierl, execute the following commands:

cd src
make clean
make

This will take a while to compile. After it's done, from the current src directory,

cd ../rllab/envs
python3 squatting.py

This should open a new MuJoCo window where Cassie squats up and down. You're done!

How to install necessary tools to run RL algorithms

1. git fetch and git pull to keep your cassierl up-to-date with the current master branch.

2. Download and install Anaconda Distribution - Python 2.7 version - here. This will take a while to install.

3. cd to your $workplacePath which was already defined in your .bashrc. If you defined it right, this is the parent directory of this repository.

4. git clone https://github.com/rll/rllab.git. This will clone RL Lab's repository to your $workplacePath directory if you haven't done so already.

5. cd rllab

6. sudo python3 setup.py install. This will add rllab to your Python packages.

7. Stay where you are in the rllab repository and sudo ./scripts/setup_linux.sh.

8. Then, sudo cp -rf scripts/ /usr/local/lib/python3.5/dist-packages/rllab-0.1.0-py3.5.egg/. This will copy the scripts folder inside rllab to the rllab folder inside Python packages.

9. sudo vim /usr/local/lib/python3.5/dist-packages/rllab-0.1.0-py3.5.egg/rllab/misc/instrument.py. There is a syntax error in this file.

10. Go to line 972 and delete the trailing comma after config.AWS_EXTRA_CONFIGS.

11. Now, cd back to your cassierl repository. Then, cd rllab/envs.

12. Try source activate rllab3. This will add a (rllab3) as a prefix to your command-line prompt.

13. python3 trpo_cassie.py. When you first run this thing, it will recommend you to edit your personal config file in rllab. Then, it will terminate. This is fine. Run sudo chmod +x /usr/local/lib/python3.5/dist-packages/rllab-0.1.0-py3.5.egg/rllab/config_personal.py. This will allow rllab to execute the newly created personal config.

14. Then, try to run it again - python3 trpo_cassie.py.

There may be some errors before you get the TRPO algorithm to run. Just import every Python modules on the way. In case of the module downsample, do the following:

sudo pip3 install --upgrade https://github.com/Theano/Theano/archive/master.zip

sudo pip3 install --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip

If, in any chance, the program requires you to install the same things to your Python 2, then repeat from steps 6, but replace python3 with python2 and pip3 with pip2 in every command you execute.

If you are able to run trpo_cassie.py. It will be like this:

Let it run for a few iterations. Then Ctrl C in your terminal to turn it off. Now, to make sure it is running and logging information on your computer. Go to the rllab repository (not the one in cassierl but the one you use to install rllab module from step 6). Look for the folder data. Dig into that folder and check if there is any log file. If so, then you have set things up correctly for TRPO.

How many iteration do I need?

Note at first Cassie will fall all over the place. After about 200 iterations or more, she will be able to not falling sideway. Observe carefully, you'll see she will mangage to fall in the up-down direction perpendicular to the ground. This is progress.

However, Yathartha said we need at least 1000 to 1500 iteration for Cassie to grasp how to stand. She will eventually fall, but her standing will last much longer.

Some useful documents to understand the reinforcement learning setup (not needed for setup)

The Docs folder contains a writeup, and presentation pdf file. For the presentation with videos, this link

Please note: there might be some mistakes or changes that have been made since the report/presentation was made.