Exploring Variational Deep Q Networks
This study provides a research-ready implementation of Tang and Kucukelbir's Variational Deep Q Network, a novel approach to maximising the efficiency of exploration in complex learning environments using Variational Bayesian Inference, using the Edward PPL. Alongside reference implementations of both Traditional and Double Deep Q Networks, a small novel contribution is presented --- the Double Variational Deep Q Network, which incorporates improvements to increase the stability and robustness of inference-based learning. Finally, an evaluation and discussion of the effectiveness of these approaches is discussed in the wider context of Bayesian Deep Learning.
The full report is available here: Exploring VDQNs.
Using the Framework
These steps assume a fresh Ubuntu installation is being used. The process may deviate slightly if a different system is used. A helper script is included for installing the required dependencies on Linux; the process is near-identical for macOS.
git clone https://github.com/HarriBellThomas/VDQN.git
cd VDQN
./init.sh
source env/bin/activateThere are four main files to be aware of:
run.py--- this is the main entrypoint for running a single instance of one of the four algorithms. It accepts a number of CLI arguments for configuring the parameters it used.DQN.py--- this is the source file containing the implementations for both DQN and DDQN.VDQN.py--- this is the source file containing the implementations for both VDQN and DVDQN.drive.py--- this script is the driver used for running experiments at scale. It constructs a collection of 80 experiments, and iteratively loops through them.
The run.py script can be used as follows:
python3 run.py
--algorithm DQN|DDQN|VDQN|DVDQN \
--environment CartPole-v0 \
--episodes 200 \
--timesteps 200 \
--lossrate 1e-2