maxencefaldor / dqn

A compact and flexible DQN implementation as well as a variety of improvements leading to the rainbow agent.

Deep Q-Network

Implementation of the DQN algorithm and six independent improvements as described in the paper Rainbow: Combining Improvements in Deep Reinforcement Learning [1].

• DQN [2]
• Double DQN [3]
• Prioritized Experience Replay [4]
• Dueling Network Architecture [5]
• Multi-step Bootstrapping [6]
• Distributional RL [7]
• Noisy Networks [8]

I provide a main.py as well as a Jupyter Notebook which demonstrate how to set up, train and compare multiple agents to reproduce the results of the aforementioned paper.

Don't hesitate to modify the default hyperparameters or the code to see how your new algorithm compare to the standard ones.

Project Structure

├── README.md
├── main.py                             # Lab where agents are declared, trained and compared
├── .gitignore
├── agents
│   ├── dqn_agent.py                    # DQN agent
│   ├── ddqn_agent.py                   # Double DQN agent
│   └── rainbow_agent.py                # Rainbow ageent
├── replay_memory
│   ├── replay_buffer.py                # The standard DQN replay memory
│   ├── prioritized_replay_buffer.py    # Prioritized replay memory using a sum tree to sample
│   └── sum_tree.py                     # Sum tree implementation used by the prioritized replay memory
└── utils
    ├── network_architectures.py        # A collection of network architectures including standard, dueling, noisy or distributional
    ├── wrappers.py                     # Wrappers and utilities to create Gym environments
    └── plot.py                         # Plot utilities to display agents' performances

In wrappers.py, I also provide a clean implementation of a CartPole Swing Up environment. The pole starts hanging down and the cart must first swing the pole to an upright position before balancing it as in normal CartPole.

Instructions

First download the source code.

git clone https://github.com/maxencefaldor/dqn.git

Finally setup the environment and install dqn's dependencies

pip install -U pip
pip install -r dqn/requirements.txt

Requirements

• PyTorch
• NumPy
• Gym
• atari-py
• Matplotlib

Acknowledgements

• @openai for Baselines
• @google for a sum tree implementation
• @higgsfield for RL-Adventure

References

[1] Rainbow: Combining Improvements in Deep Reinforcement Learning, Hessel et al., 2017.
[2] Playing Atari with Deep Reinforcement Learning, Mnih et al., 2013.
[3] Deep Reinforcement Learning with Double Q-learning, Hasselt et al., 2015.
[4] Prioritized Experience Replay, Schaul et al., 2015.
[5] Dueling Network Architectures for Deep Reinforcement Learning, Wang et al., 2015.
[6] Reinforcement Learning: An Introduction, Sutton and Barto, 1998.
[7] A Distributional Perspective on Reinforcement Learning, Bellemare et al., 2017.
[8] Noisy Networks for Exploration, Fortunato et al., 2017.