LegOpt Benchmark

This repository is for the LegOpt Benchmark (Legged-robot posture and trajectory Optimization Benchmark).

LegOpt is a benchmark for evaluating different algorithms at the tasks of posture generation and trajectory optimization on legged robots. There are 50 problems for each task, which are associated with a pre-defined robot model, scenario, constraints and cost functions.

The scenario is from the Destruction Scenarios Dataset.

In this repository we provide:

latest benchmark results
problem definition data
solution evaluation scripts
usage and submission instructions

Paper

If you use this in your research, please cite:

Martim Brandao, Kenji Hashimoto, and Atsuo Takanishi, "SGD for robot motion? The effectiveness of stochastic optimization on a new benchmark for biped locomotion tasks", in 17th IEEE-RAS International Conference on Humanoid Robots, 2017.

Make sure to check the paper for details (PDF here).

Latest benchmark results

			Posture generation			Trajectory optimization
Method	Multi starts	Sample size	Success rate	Cost	Time (s)	Success rate	Cost	Time (s)
SQP	0	100	49/50	426.68	1.32	19/50	0.13	0.63
SGD	0	100	44/50	719.83	3.76	4 /50	0.20	1.38
Adam	0	100	42/50	606.38	4.15	18/50	0.14	1.22
Nadam	0	100	39/50	590.26	2.83	16/50	0.14	0.81
SQP	10	100	50/50	426.64	1.44	42/50	0.13	0.62
SGD	10	100	49/50	797.55	6.30	27/50	0.17	2.17
Adam	10	100	49/50	642.92	5.97	31/50	0.14	1.50
Nadam	10	100	49/50	641.97	5.67	35/50	0.15	1.19
SQP	10	80	16/50	297.75	2.74	0 /50	0.00	0.00
SGD	10	80	50/50	788.22	5.82	26/50	0.16	1.99
Adam	10	80	50/50	628.55	4.18	35/50	0.14	2.05
Nadam	10	80	50/50	619.66	2.90	33/50	0.14	1.10
I-SQP	10	80	50/50	460.58	1.00	42/50	0.13	0.53
I-SGD	10	40	49/50	833.01	4.13	30/50	0.15	1.30
I-Adam	10	40	50/50	660.53	3.27	40/50	0.14	1.07
I-Nadam	10	40	50/50	666.94	2.66	41/50	0.15	0.73

Note: costs and times are averaged over successfully solved problems.

Latest benchmark results (from "tough" initialization in collision)

			Posture generation			Trajectory optimization
Method	Multi starts	Sample size	Success rate	Cost	Time (s)	Success rate	Cost	Time (s)
SQP	0	100	43/50	415.62	1.43	11/50	0.12	0.67
SGD	0	100	41/50	769.82	3.55	2 /50	0.18	1.26
Adam	0	100	46/50	616.37	3.78	2 /50	0.14	1.27
Nadam	0	100	48/50	645.27	2.33	1 /50	0.11	0.92
SQP	10	100	50/50	429.45	1.74	29/50	0.12	0.84
SGD	10	100	48/50	783.34	5.11	29/50	0.16	3.04
Adam	10	100	50/50	671.93	4.30	32/50	0.15	2.10
Nadam	10	100	50/50	678.50	3.20	30/50	0.15	1.96
SQP	10	80	20/50	278.90	4.00	0 /50	0.00	0.00
SGD	10	80	49/50	774.82	4.44	30/50	0.16	2.49
Adam	10	80	50/50	642.19	3.91	36/50	0.15	2.12
Nadam	10	80	50/50	637.78	2.70	34/50	0.14	1.47
I-SQP	10	80	50/50	468.25	1.34	38/50	0.14	0.71
I-SGD	10	40	49/50	809.90	5.17	30/50	0.15	1.40
I-Adam	10	40	50/50	668.34	3.66	34/50	0.15	1.34
I-Nadam	10	40	50/50	684.47	3.44	28/50	0.14	1.64

Note: costs and times are averaged over successfully solved problems.

Data format

JSON data for problems

Problems for each task are defined on JSON files. Please have a look inside folder data.

Maybe more intuitive than writing the schema here is to show how you would access the information in C++ or python. So here it is:

jsonroot["robot"]                         # name of the robot model used (this is always "atlas" for now)
jsonroot["scenario"]                      # name of the Destruction Scenario used (this is always "garage_easier" for now)
jsonroot["task"]                          # either "posture" or "motion", for posture generation or trajectory optimization respectively
jsonroot["problems"][p]                   # problem number p
jsonroot["problems"][p]["id"]             # the id of problem number p
jsonroot["problems"][p]["definition"]     # the definition of problem number p
jsonroot["problems"][p]["definition"][d]  # a string representing stance number d of problem number p

The format for strings representing stances is the following:

"<name-of-link-in-contact>,<space-separated-quaternion-xyzw> <space-separated-translation-xyz>,<name-of-link-in-contact>,<space-separated-quaternion-xyzw>,"

It assumes a rotation and translation from the world reference frame.

Example 1 (only the left foot is in contact):

"l_foot,-0.4896072729 -0.1111596603 -0.0645970827 -0.8624125841 -12.53751087 0.6949124932 5.57654953 ,"

Example 2 (both feet in contact):

"l_foot,-0.4896072729 -0.1111596603 -0.0645970827 -0.8624125841 -12.53751087 0.6949124932 5.57654953 ,r_foot,-0.4893193662 -0.1129687271 -0.06397458697 -0.8623873125 -12.60409355 1.338053823 5.595780373 ,"

JSON data for solutions

In order to evaluate your solutions with the scripts we provide, you should add them to a JSON file in the following way.

You should start from the respective problems JSON and add a "solution" object to each problem that you solve.

jsonroot["problems"][p]["solution"]                 # holds your solutions for problem number p

You can solve each problem with several methods, so "solution" is a list where each item contains the solution vector, the name of the method used and the computation time.

jsonroot["problems"][p]["solution"][s]["method"]    # name of the method used to obtain solution s
jsonroot["problems"][p]["solution"][s]["timeSec"]   # computation time (in seconds) to obtain solution s
jsonroot["problems"][p]["solution"][s]["x"][f]      # a string representing the full-body configuration number f of the solution

The format for strings representing a full-body configuration are "\n"-separated values of the DOFs (joints followed by translation and then quaternion). Basically, what you would obtain in C++ with operator<< on a Eigen::VectorXd.

"<joint1value>\n<joint2value>\n(...)<jointNvalue>\n<x>\n<y>\n<z>\n<qx>\n<qy>\n<qz>\n<qw>"

Example:

"0.0750938\n-0.0115883\n0.0618958\n0.174954\n-1.39626\n3.05126\n0.365756\n-0.442819\n-0.0552547\n-0.610865\n0.13954\n0.169922\n1.39626\n3.07882\n-0.31723\n-0.0876133\n-0.072731\n-0.201933\n0.0678042\n-0.0458344\n0.750349\n-0.783705\n0.208692\n-0.211367\n-0.0466808\n-0.786813\n0.742743\n-0.0395884\n0.324678\n-12.55\n0.949195\n6.42098\n0.448645\n0.00514189\n-0.0392512\n0.997566"

Usage and submission instructions

Clone this repository

Clone this repository using

git clone --recursive https://github.com/martimbrandao/legopt-benchmark.git

Dependencies

The list of dependencies is:

Python 2.7
numpy
scipy
JSON
OpenRAVE (we tested on commit 0a05f794a5901885a25ed5cbd6e70eb3f37e0edd)

If you don't have them yet do:

sudo apt-get install python python-numpy python-scipy

Clone, build and install OpenRAVE. You can follow these instructions if you are unsure how to do so.

Downloading the additional data

If you just want the problem definition data, then the JSON files are already inside the data folder. However, if you would like to compare the performance of your algorithms using the same problem initializations as I did, please use these files. The files are equivalent to the ones under the data folder, except for additional x0 and x0hard variables on each problem (same format as the solution variable x).

If you would like to try and run the scripts on my solutions, you can download them from here, and place the contents (a folder called solutions12-full-with-easy-and-hard-init) inside solutions/.

Evaluating your benchmark solutions

Run:

python LegOptEvalFast.py <path-to-solution-json-file> --verbose

You can also add a --interactive argument to visualize the solutions as they are checked in OpenRAVE.

If you would like to try and run the script with my solutions, see Downloading the additional data above.

Adding your results to this page

Contact me and I will add your results here.

For normal evaluations you are not required to initialize your algorithms from the same staring point x0 as provided in the additional data. However, if you want to evaluate your algorithm under tough initializations then you should initialize your algorithms using the x0hard values provided in the additional data (see Downloading the additional data above).

About

Legged-robot posture and trajectory Optimization Benchmark

Languages

Language:Python 100.0%