Reversi reinforcement learning by AlphaGo Zero methods.
@mokemokechicken's training hisotry is Challenge History.
- Python 3.6.3
- tensorflow-gpu: 1.3.0
- tensorflow==1.3.0 is also ok, but very slow. When
play_gui
, tensorflow(cpu) is enough speed.
- tensorflow==1.3.0 is also ok, but very slow. When
- Keras: 2.0.8
This AlphaGo Zero implementation consists of three worker self
, opt
and eval
.
self
is Self-Play to generate training data by self-play using BestModel.opt
is Trainer to train model, and generate next-generation models.eval
is Evaluator to evaluate whether the next-generation model is better than BestModel. If better, replace BestModel.- If
config.play.use_newest_next_generation_model = True
, this worker is useless. (It is AlphaZero method)
- If
For evaluation, you can play reversi with the BestModel.
play_gui
is Play Game vs BestModel using wxPython.
data/model/model_best_*
: BestModel.data/model/next_generation/*
: next-generation models.data/play_data/play_*.json
: generated training data.logs/main.log
: log file.
If you want to train the model from the beginning, delete the above directories.
pip install -r requirements.txt
cp requirements.txt conda-requirements.txt
- Comment out lines for
jedi
,Keras
,parso
,python-dotenv
,tensorflow-tensorboard
,wxPython
libraries - Replace '-' with '_' for
ipython-genutils
,jupyter-*
,prompt-toolkit
libraries
conda env create -f environment.yml
source activate reversi-a0
conda install --yes --file conda-requirements.txt
If you want use GPU,
pip install tensorflow-gpu
Create .env
file and write this.
KERAS_BACKEND=tensorflow
Download trained BestModel(trained by bellow Challenge 1) for example.
sh ./download_best_model.sh
Download trained the newest model(trained by Challenge 2, 3) as BestModel.
sh ./download_model.sh <version>
ex)
sh ./download_model.sh 2
I think the main difference between 'AlphaGo Zero' and 'AlphaZero' is whether using eval
or not.
It is able to change these methods by configuration.
PlayConfig#use_newest_next_generation_model = False
PlayWithHumanConfig#use_newest_next_generation_model = False
- Execute
Evaluator
to select the best model.
PlayConfig#use_newest_next_generation_model = True
PlayWithHumanConfig#use_newest_next_generation_model = True
- Not use
Evaluator
(the newest model is selected asself-play
's model)
In DeepMind's paper, it seems that policy(π) data saved by self-play are distribution in proportion to pow(N, 1/tau). After the middle of the game, the tau becomes 0, so the distribution is one-hot.
PlayDataConfig#save_policy_of_tau_1 = True
means that the saved policy's tau is always 1.
If you find a good parameter set, please share in the github issues!
nb_game_in_file,max_file_num
: The max game number of training data isnb_game_in_file * max_file_num
.
simulation_num_per_move
: MCTS number per move.c_puct
: balance parameter of value network and policy network in MCTS.resign_threshold
: resign thresholdparallel_search_num
: balance parameter(?) of speed and accuracy in MCTS.prediction_queue_size
should be same or greater thanparallel_search_num
.
dirichlet_alpha
: random parameter in self-play.
For training model, execute Self-Play
, Trainer
and Evaluator
.
python src/reversi_zero/run.py self
When executed, Self-Play will start using BestModel. If the BestModel does not exist, new random model will be created and become BestModel.
--new
: create new BestModel--type mini
: use mini config for testing, (seesrc/reversi_zero/configs/mini.py
)
python src/reversi_zero/run.py opt
When executed, Training will start. A base model will be loaded from latest saved next-generation model. If not existed, BestModel is used. Trained model will be saved every 2000 steps(mini-batch) after epoch.
--type mini
: use mini config for testing, (seesrc/reversi_zero/configs/mini.py
)--total-step
: specify total step(mini-batch) numbers. The total step affects learning rate of training.
python src/reversi_zero/run.py eval
When executed, Evaluation will start. It evaluates BestModel and the latest next-generation model by playing about 200 games. If next-generation model wins, it becomes BestModel.
--type mini
: use mini config for testing, (seesrc/reversi_zero/configs/mini.py
)
python src/reversi_zero/run.py play_gui
play_gui
uses wxPython
.
It can not execute if your python environment is built without Framework.
Try following pyenv install option.
env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.6.3
For Anaconda users:
conda install python.app
pythonw src/reversi_zero/run.py play_gui
When executed, ordinary reversi board will be displayed and you can play against BestModel. After BestModel moves, numbers are displayed on the board.
- Top left numbers(1) mean 'Visit Count (=N(s,a))' of the last search.
- Bottom left numbers(2) mean 'Q Value (=Q(s,a)) on AI side' of the last state and move. The Q values are multiplied by 100.
NBoard is a very good reversi GUI and has strong reversi engines, which runs on Windows, Mac, and Linux (JRE required).
It can add external engines that implement NBoard Protocol.
- (0) launch NBoard from command line(need environment variables like PATH)
- ex)
java -jar /Applications/NBoard/nboard-2.0.jar
- ex)
- (1) select menu
Engine -> Select Opponent...
- (2) clike button
Add Engine
- (3) set parameter:
Name
=RAZ
(for example)Working Directory
= PATH TO THIS PROJECTCommand
=nboard_engine
orbash nboard_engine
- (4) Engine Level N is set as
simulation_num_per_move=N*20
NBoard cannot play with two different engines (maybe). However, it can select different engines of play-engine and analysis-engine.
So, convenient way to evaluate your model is for example,
- select this engine as play-engine (or analysis-engine), another engine as analysis-engine (or play-engine).
- check menu
View -> Highlight Best Move
- start
User plays Black
(or White) - You simply choose the best move of analysis-engine.
I have little confidence about hint
protocol as analysis-engine (there is odd behavior),
but work in my environment.
pip install tensorboard
tensorboard --logdir logs/tensorboard/
And access http://<The Machine IP>:6006/
.
If you can not launch tensorboard by error,
try to create another new plain project which includes only tensorflow
and tensorboard
.
And
tensorboard --logdir <PATH TO REVERSI DIR>/logs/tensorboard/
In my environment of GeForce GTX 1080, memory is about 8GB, so sometimes lack of memory happen.
Usually the lack of memory cause warnings, not error.
If error happens, try to change per_process_gpu_memory_fraction
in src/worker/{evaluate.py,optimize.py,self_play.py}
,
tf_util.set_session_config(per_process_gpu_memory_fraction=0.2)
Less batch_size will reduce memory usage of opt
.
Try to change TrainerConfig#batch_size
in NormalConfig
.
- CPU: 8 core i7-7700K CPU @ 4.20GHz
- GPU: GeForce GTX 1080
- 1 game in Self-Play: about 10~20 sec (simulation_num_per_move = 100, thinking_loop = 1).
- 1 step(mini-batch, batch size=512) in Training: about 1.8 sec.