DRL - PPO - Soccer Twos
Udacity Deep Reinforcement Learning Nanodegree Program
Observations:
- To run the project just execute the main.py file.
- If you are not using a windows environment, you will need to download the corresponding "Soccer" version for you OS system. Mail me if you need more details about the environment .exe file.
- The checkpoint.pth has the expected average score already hit.
Requeriments:
- tensorflow: 1.7.1
- Pillow: 4.2.1
- matplotlib
- numpy: 1.11.0
- pytest: 3.2.2
- docopt
- pyyaml
- protobuf: 3.5.2
- grpcio: 1.11.0
- torch: 0.4.1
- pandas
- scipy
- ipykernel
- jupyter: 5.6.0
The problem:
- The task involves a soccer game with 2 teams, each one having 2 players: 1 striker and one 1 keeper.
- There is no goal defined for default, so I decided to train against a random team until my agents archive a score of 95 wins into 100 games.
- The goalies have 4 actions.
- The strikers have 6 actions.
The solution:
- The biggest problem in this scenario is to control the exploration vs. exploitation rate. I tried approaches such as Double DQN with an exponential exploration rating decay as well as the DDPG approach with prioritized replay experience for diversification of the experiences on learning, but I couldn't find the right configuration for the hyperparameters that could make the agents converge.
- So I changed the approach for a PPO strategy since this kind of method is easier to configure and controls the exploration very well by itself using probabilistic decisions. After a lot of different implementations, I've reached the current solution.
- There are still some items to improve such as convergence time and multi teams training, but I'm satisfied with the current results. On my last test, I could archive the goal (95 wins in 100 games) with a little more than 5000 episodes and I consider that a great result if I look back to all the tries I made before.
- It was really good for my learning as I had not used PPO approaches at this level before trying this environment and for sure the knowledge acquired here will be very relevant for my next projects.
- Talking about the implementation, it has an actor critic neural model and is using a proximal policy optimization learning function with the trusted region approach. The learning happens after each episode (controlled by the environment), and it uses mini-batches from the episode experiences after the reward calculation using the N-Step method that combines the temporal difference discount with monte carlo tree search exploration (in this case the N-Step range is the role episode).
- For now, I'll try other variations changing when the learning happens and using multi teams for experience gathering. I hope I can archive superhuman results with 5000 episodes or less (the agents are good but not super humans with 5000 episodes).
- One last consideration. To beat a random team looks easier at the beginning, but if you consider that random agents win 1/3 of the games and the draw rate of random games is 1/3, the AI has overcome a big challenge reaching a 95% win rate. It's incredible how a random agent can score with just a few steps.
The hyperparameters:
-
The file with the hyperparameters configuration is the main.py.
-
If you want you can change the model configuration to into the model.py file.
-
The actual configuration of the hyperparameters is:
- Learning Rate Goalie: 8e-5
- Learning Rate Striker: 1e-4
- Gamma: 0.995
- Batch Size: 32
- Epsilon: 0.1
- Entropy Weight: 0.001
-
For the neural models:
-
Actor
- Hidden: (input, 256) - ReLU
- Hidden: (256, 128) - ReLU
- Output: (128, action_size) - Softmax
-
Critic
- Hidden: (input, 256) - ReLU
- Hidden: (256, 128) - ReLU
- Output: (128, 1) - Linear
-