filippogiruzzi / reinforcement_learning_resources

Personal Deep Reinforcement Learning class notes

Reinforcement Learning resources

Keywords: Deep Reinforcement Learning, UC Berkeley

This repository contains some personal Deep Reinforcement Learning class notes in .pdf format.

Table of contents

1. Class notes content
   1.1 CS285 (UC Berkeley) - Deep Reinforcement Learning by Sergey Levine
2. Resources

1. Class notes content

1.1 CS285 (UC Berkeley) - Deep Reinforcement Learning by Sergey Levine

These notes summarize the main Reinforcement Learning algorithms, both in theory and in practice with some tips & hacks for efficient implementation.

1. Introduction

WIP

2. Supervised Learning of behaviors

1. Goal
2. Algorithms
   2.1 DAgger: Dataset Aggregation
3. Tips & hacks

3. Introduction to Reinforcement Learning

1. Goal
2. Algorithms
   2.1 Global structure
   2.2 Exemples

4. Policy Gradients

1. Algorithms
   1.1. REINFORCE
2. Tips & hacks

5. Actor-Critic algorithms

1. Algorithms
   1.1. Batch Actor-Critic
   1.2. Online Actor-Critic
2. Tips & hacks

6. Value function methods

1. Algorithms
   1.1. Policy iteration
   1.2. Policy iteration with Dynamic programming
   1.3. Value iteration
   1.4. Fitted Value iteration
   1.5. Fitted Q-iteration
   1.6. Online Q-iteration
2. Tips & hacks

7. Deep Reinforcement Learning with Q-functions

1. Algorithms
   1.1. Q-learning with replay buffer
   1.2. Q-learning with replay buffer and target network
   1.3. DQN: classic Deep Q-learning
   1.4. DDPG: Q-learning for continuous actions
2. Tips & hacks

8. Advanced Policy Gradients

WIP

9. Model-based planning

WIP

10. Model-based Reinforcement Learning

1. Algorithms
   1.1. Model-based Reinforcement Learning version 0.5
   1.2. Model-based Reinforcement Learning version 1.0
   1.3. Model-based Reinforcement Learning version 1.5
   1.4. Model-based Reinforcement Learning with latent space models
2. Tips & hacks

11. Model-based Policy Learning

1. Algorithms
   1.1. Model-based Reinforcement Learning version 2.0
   1.2. DYNA: online Q-learning model-free Reinforcement Learning with a model
   1.3. General DYNA-style model-based Reinforcement Learning
   1.4. MBA: Model-based Acceleration – MVE: Model-based Value Ex- pansion – MBPO: Model-based Policy Optimization
   1.5. Divide and Conquer Reinforcement Learning
2. Tips & hacks

12. Variational Inference & Generative models

WIP

13. Control as inference

1. Algorithms
   1.1 Soft Q-learning

14. Inverse Reinforcement Learning

1. Algorithms
   1.1. Maximum Entropy Inverse Reinforcement Learning
2. Tips & hacks

15. Transfer & Multi-task Learning

1. Tips & hacks

16. Distributed Reinforcement Learning

WIP

17. Exploration

1. Algorithms
   1.1. Pre-train & finetune
2. Tips & hacks

18. Meta-learning

WIP

19. Information theory

WIP

2. Resources

These notes were widely inpspired by:

• CS285 (UC Berkeley) - Deep Reinforcement Learning