Multiple policy heads in RNaD
andreipauliuc-ads opened this issue · comments
The RNaD paper mentions using a neural network with 3 policy heads, one for initial piece deployment and two for the play phase (one for selecting a piece and one for moving that piece). The example code for RNaD presents the simple case of a network with only one policy head.
How do you deal with multiple policy heads in V-Trace algorithm and to compute Update_policy part?
I believe we have already answered this. Can you check if #1053 answers your question?
@lanctot thanks for the reply and pointing to the past discussions.
It partialy helps us. But we are in the same situation presented in the last comment in #1053; we have multiple prediction heads in order to deconstruct the wide action space: there's one head for the specific action type, one which selects the unit that should do the action (some actions require a unit, others don't), one which selects a target unit and location, etc.; there are a total of 5 heads where the output from one head is used as input for another head and so on. This is what leads to confusion when applying V-Trace to this type of deconstructed policy.
To relate to RNaD in Stratego, it's clear you use one head in each specific phase of the game and only one value head. But you also have the case where you select a piece based on one policy head and then use its output to determine the piece destination from another policy head. How is this reflected in the game trajectory? Are there two consecutive steps each belonging to a policy head but for the same player?
Hi,
You got it right, there are just two consecutive steps for the same player (one to select the piece and one to move it) during the gameplay phase. The head is then selected accordingly.
Hope this helps,
Julien