Chapter6 DQN Pong can't calculate loss

[Pedro Oliveira]

I'm running the exact same code yet i get this error:

C:\Users\Utilizador\anaconda3\python.exe "C:/Users/Utilizador/Thesis/Deep Reinforcement Learning Hands-On/Chapter6/02_dqn_pong.py"
DQN(
(conv): Sequential(
(0): Conv2d(4, 32, kernel_size=(8, 8), stride=(4, 4))
(1): ReLU()
(2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2))
(3): ReLU()
(4): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))
(5): ReLU()
)
(fc): Sequential(
(0): Linear(in_features=3136, out_features=512, bias=True)
(1): ReLU()
(2): Linear(in_features=512, out_features=6, bias=True)
)
)
880: done 1 games, mean reward -21.000, eps 0.99, speed 720.31 f/s
1848: done 2 games, mean reward -21.000, eps 0.98, speed 659.82 f/s
2670: done 3 games, mean reward -21.000, eps 0.97, speed 642.90 f/s
3492: done 4 games, mean reward -21.000, eps 0.97, speed 666.83 f/s
4659: done 5 games, mean reward -20.600, eps 0.95, speed 671.71 f/s
Best mean reward updated -21.000 -> -20.600, model saved
5765: done 6 games, mean reward -20.167, eps 0.94, speed 629.02 f/s
Best mean reward updated -20.600 -> -20.167, model saved
6682: done 7 games, mean reward -20.143, eps 0.93, speed 645.23 f/s
Best mean reward updated -20.167 -> -20.143, model saved
7668: done 8 games, mean reward -20.125, eps 0.92, speed 628.90 f/s
Best mean reward updated -20.143 -> -20.125, model saved
8648: done 9 games, mean reward -20.000, eps 0.91, speed 606.56 f/s
Best mean reward updated -20.125 -> -20.000, model saved
9919: done 10 games, mean reward -19.700, eps 0.90, speed 623.48 f/s
Best mean reward updated -20.000 -> -19.700, model saved
Traceback (most recent call last):
File "C:/Users/Utilizador/Thesis/Deep Reinforcement Learning Hands-On/Chapter6/02_dqn_pong.py", line 169, in
loss_t = calc_loss(batch, net, tgt_net, device=device)
File "C:/Users/Utilizador/Thesis/Deep Reinforcement Learning Hands-On/Chapter6/02_dqn_pong.py", line 96, in calc_loss
state_action_values = net(states_v).gather(1, actions_v.unsqueeze(-1)).squeeze(-1)
RuntimeError: gather_out_cpu(): Expected dtype int64 for index

Process finished with exit code 1

I'm relatively new to Python and couldn't find any solution to this kind of problem on other sources. Anyone had this same problem or knows how to fix it?

[Pedro Oliveira]

I just found that if you're on newer versions of PyTorch you should define your calc_loss like this:

`def calc_loss(batch, net, tgt_net, device="cpu"):
states, actions, rewards, dones, next_states = batch

states_v = torch.tensor(states).to(device)
next_states_v = torch.tensor(next_states).to(device)
actions_v = torch.tensor(actions).to(device, dtype=torch.int64)
rewards_v = torch.tensor(rewards).to(device)
done_mask = torch.tensor(dones).to(device, dtype=torch.bool)

state_action_values = net(states_v).gather(1, actions_v.unsqueeze(-1)).squeeze(-1)
next_state_values = tgt_net(next_states_v).max(1)[0]
next_state_values[done_mask] = 0.0
next_state_values = next_state_values.detach()

expected_state_action_values = next_state_values * GAMMA + rewards_v
return nn.MSELoss()(state_action_values, expected_state_action_values)`

The only differences between this and the code in the book are the declarations of vars actions_v and done_mask