Safe Reinforcement Learning Algorithms

HCOPE
Safe Exploration
Off Policy Evaluation
Solving side effects

HCOPE (High-Confidence Off-Policy Evaluation.)

Python Implementation of HCOPE lower bound evaluation as given in the paper: Thomas, Philip S., Georgios Theocharous, and Mohammad Ghavamzadeh. "High-Confidence Off-Policy Evaluation." AAAI. 2015.

Requirements

PyTorch
Numpy
Matplotlib
scipy
gym

Running Instructions

Modify the environment in the main function, choosing from OpenAI gym. (Currently the code works for discrete action spaces)
Run python hcope.py

Notes

The file policies.py contains the policy used in the code. Modify the policy to suit your needs in this file.
To reproduce the graph given in the original paper explaining the long tail problem of importance sampling, use the

visualize_IS_distribution()

method. Also, a graph of distribution of Importance sampling ratio is created which nicely explains the high variance of the simple IS estimator.

All the values required for offpolicy estimation are initialized in the HCOPE class initialization.
Currently the estimator policy is defined as a gaussian noise(mean,std_dev) added to the behavior policy for estimator policy initialization in the function setup_e_policy(). The example in paper uses policies differing by natural gradient. But, this works as well.
To estimate c*, I use the BFGS method which does not require computing hessian or first order derivative.
The hcope_estimator() method also implements a sanity check, by computing the discriminant of the quadratic in parameter delta(confidence). If it does not satisfy the basic constraints, the program prints the bound predicted is of zero confidence.
The random variables are implemented using simple importance sampling. Per-decision importance sampling might lead to better bounds and is to be explored.
A bilayer MLP policy is used for general problems.
Results:
Output format:

Safe exploration in continuous action spaces.

Paper: Safe Exploration in Continuous Action Spaces - Dalal et al.

Running Instructions

Go inside safe_exploration folder
First learn the safety function by collecting experiences
python learn_safety_function.py
Now using the learned safety function, add the path of these learned torch weights in the train_safe_explorer.py file. After that:
python train_safe_explorer.py
This enables agent to learn while following the safety constraints.