llucid-97 / muzero-general

MuZero

MuZero General: WINDOWS PORT

This is a windows port of Duvaud et al's MuZero implementation

Implementation Changes:

I have used torch multiprocessing instead of ray, and made a few sacrifices of correctness for speed in implementing the asynchronous communications between processes.a

In general, where ray requests a response from another process, I have made the processes just queue answers instead. This means the responses are slightly stale. This means I don't reprodue exactly the same results, but the performance on par.

API Changes:

• shared storage is deprecated in favour of setting up queues in the config. As such it's easiest to just inherit from the base config class when creating new games

I do not plan on updating this with their new features. please use their official repo

A commented and documented implementation of MuZero based on the Google DeepMind paper and the associated pseudocode. It is designed to be easily adaptable for every games or reinforcement learning environments (like gym). You only need to edit the game file with the parameters and the game class. Please refer to the documentation and the example.

MuZero is a model based reinforcement learning algorithm, successor of AlphaZero. It learns to master games without knowing the rules. It only knows actions and then learn to play and master the game. It is at least more efficient than similar algorithms like AlphaZero, SimPLe and World Models. See How it works

Features

• Fully connected network in PyTorch
• Multi-Threaded with Ray
• CPU/GPU support
• TensorBoard real-time monitoring
• Single and multiplayer mode
• Commented and documented
• Easily adaptable for new games
• Examples of board and Gym games (See list below)
• Pretrained weights available
• Play against MuZero mode with policy and value tracking
• Residual Network
• Atari games

Games already implemented with pretrained network available

• Cartpole
• Lunar Lander
• Connect4

Demo

All performances are tracked and displayed in real time in tensorboard :

Testing Lunar Lander :

Getting started

Installation

cd muzero-general
pip install -r requirements.txt

Training

Edit the end of muzero.py:

muzero = Muzero("cartpole")
muzero.train()

Then run:

python muzero.py

To visualize the training results, run in a new terminal:

tensorboard --logdir ./

Testing

Edit the end of muzero.py:

muzero = Muzero("cartpole")
muzero.load_model()
muzero.test()

Then run:

python muzero.py

Authors

• Werner Duvaud
• Aurèle Hainaut
• Paul Lenoir