Unity Machine Learning Agent

Introduction

This is repository for Unity Machine Learning Agent and Reinforcement Learning(RL).

Unity release awesome tool for making reinforcement learning environment! Unity Machine Learning

Some of the environment made by purchased models. In this case, it is hard to provide unity codes. However, there are some simple environments which are made by simple or free model. I will provide unity codes for those environments.

Environment

Software

Windows10 (64bit), Ubuntu16.04
Python 3.6.5
Anaconda 4.2.0
Tensorflow-gpu 1.12.0
Unity version 2017.2.0f3 Personal
Unity ML-Agents: 0.8.1

Hardware

CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHZ
GPU: GeForce GTX 1080Ti
Memory: 8GB

Unity ML Environments

Deep Q Learning Based High Level Driving Policy Determination

This project was one of the project of Machine Learning Camp Jeju 2017.

After the camp, simulator was changed to Unity ML-Agents.

Published papers using the environment

The Repository link of this project as follows.

Repository Link!!!

Vehicle Environment(Dynamic Obstacles)

The agent of this environment is vehicle. Obstacles are other 8 different kind of vehicles. If vehicle hits other vehicle, it gets minus reward and game restarts. If vehicle hits start, it gets plus reward and game goes on. The specific description of the environment is as follows.

- State: Game View (80x80x1 grayscale image)
- Action: 3 Actions (Left, Right, stay)
- Reward 
	- Driving at the center of the lane: +1 (linearly decrease)
	- Collide with other vehicles (-10)

Demo video: youtube link

Above demo, referenced papers to implement algorithm are as follows.

Dueling Network Architectures for Deep Reinforcement Learning

Vehicle Environment(Static Obstacles)

This environment won ML-Agents Challenge!!! 👑

The agent of this environment is vehicle. Obstacles are static tire barriers. If vehicle hits obstacle, it gets minus reward and game restarts. If vehicle hits start, it gets plus reward and game goes on. The specific description of the environment is as follows.

Specific description of the environment

Demo video: youtube link

Above demo, referenced papers to implement algorithm are as follows.

Breakout

This is breakout environment, which is popular environment for testing RL algorithm.

The red bar (agent) moves left and right side to hit the ball. If the ball collides with a block, it breaks the block. In every episode, ball is fired in random direction.

The rules of the breakout are as follows.
- Visual Observation: 80x80x1 grayscale image  
- Actions: 3 actions (left, stay, right)

+1 Reward
- If ball breaks a block, the agent gets +1 reward 

-1 Reward
- If the agent misses the ball, the agent gets -1 reward

Terminal conditions
- If the agent misses the ball 
- If the agent breaks all the blocks

Pong

This is simple and popular environment for testing deep reinforcement learning algorithms.

Two bars have to hit the ball to win the game. In my environment, left bar is agent and right bar is enemy. Enemy is invincible, so it can hit every ball. In every episode, ball is fired in random direction.

The rules of the pong are as follows.
- Visual Observation: 40x80x1 grayscale image 
- Vector Observation: 32 
- Actions: 3 actions (up, stay, right)

- Agent hits the ball: reward +0.5
- Agent misses the ball: reward -1

Terminal conditions
- Agent misses the ball
- After 1000 time steps, the episode ends

Predator Prey (MultiAgent)

This is popular environment for testing Multi Agent deep reinforcement learning algorithms. The lions and a sheep are agents. Lions are predator, so they have to capture the sheep. Sheep is prey, so it has to run away from the lions.

Number of the Lions can be changed from 1 to 6 using the following python code

The rules of the Predator Prey are as follows.
- Visual Observation: 80x80x3 image 
- Vector Observation: 3 -> x position, z position, role(0: Prey, 1: Predator) 
- Actions: 4 actions (up, down, left, right)

Reward (Predator)
- Every move: -0.01 
- Predator captures the prey: +1 

Reward (Prey)
- Every move: +0.01
- Predator captures the prey: -1

Terminal conditions
- Predator captures the prey 
- After 500 steps

Hopper

This is popular environment for testing Continuous Action deep reinforcement learning algorithms. The agent has to move to the right side.

The rules of the Predator Prey are as follows.
- Vector Observation: 19 * 4 (19 data * 4 stacks) 
	- foot: x distance/50, local position (x,y,z), velocity (x, y), angular velocity (z)
	- leg1: local position (x,y,z), velocity (x,y), angular velocity (z)
	- leg2: local position (x,y,z), velocity (x,y), angular velocity (z)
- Actions: 3 Continuous actions (-1 ~ 1) -> Torque(foot, leg1, leg2)

Reward 
- y position of leg2 < 0.8: -1
- x potision of foot > 50: +1
- (0.01 * foot x velocity) + (0.00001 * foot x distance) 

Terminal conditions
- y position of leg2 < 0.8
- x potision of foot > 50

Kyushik / Unity_ML_Agent