Calculating Success Rate for Physical Deception

[Zishun Yu]

Hello, I am trying to reproduce the experiments on physical deception scenario. Do you mind if I am asking the setup for calculating the success rate.
I. What is the distance threshold?
II. In which situation an agent succeed?
a. as long as it was close enough to the target at any time step
b. it was close enough to target at end of an episode
c. it should be close to target for a while, multiple time steps
d. any other situation?
III. How to calculate the rate if we have multiple good/adv agents?
a. calculate the average rate
b. pick the minimum distance to target (under setup II. ) of good/adv agents at each episode
c. any other situation?

Thanks for your time.

[Ryan Lowe]

Hi! Thanks for the question. I forget the exact distance threshold (I apologize, I realize this is my fault and it hurts replication), but it was in a small-to-medium sized ball around the target landmark (we calibrated it visually, since the agents experience some action noise).

For question II, the answer is b) -- we calculated based on the closeness to the target at the end of the episode.

For III, the answer is also b) --- only 1 good agent has to be close to the target landmark, so we look at the minimum over agents. Usually we only have 1 adversary, so this is not a concern on that side, but if you have more you can also take the minimum.