Missing Agents in SimulationScene

[Leon De Andrade]

Hi,

I experimented with the following example: https://github.com/nvr-avg/trajdata/blob/main/examples/simple_sim_example.py

This worked out of the box, but with some different datasets / settings it didn't.

1. Same Dataset, but freeze_agents=False

KeyError: ('4f41f55039c34783b0f63a3ebc196434', 1)

2. eupeds_eth Dataset with freeze_agents=False

KeyError: ('2', 2)

3. eupeds_eth Dataset with freeze_agents=True

KeyError: 'labels [5] not found in level'

I'm not positing the full stack trace as it probably isn't very helpful. Best to reproduce it quickly on a local machine.

I figured the issue might be in https://github.com/nvr-avg/trajdata/blob/24251636bc708bd20e91a538b68095300a3199c8/src/trajdata/simulation/sim_df_cache.py#L125.

As far as I understand it, it drops the available data and replaces it with the calculated data. The problem is that the calculated data only contains the agents that have been in the scene before. Thus, data from agents that appear new in the scene us just dropped without any replacement.

I wrote some quick code to replace the above (L124 - 127) with:

self.persistent_data_df = pd.concat([self.persistent_data_df, sim_step_df])
self.persistent_data_df = self.persistent_data_df[~self.persistent_data_df .index.duplicated(keep='last')]

This will just concatenate the simulated data and then remove the duplicated. Data from new agents in the scene is therefore kept.

This seemed to fix all three cases above. Which is a little bit surprising, because I assumed it wouldn't fix 3 as the agents should be frozen.

Did I understand the code correctly? Perhaps you can have look. Thanks!

[Boris Ivanovic]

Hi @Leon0402 , thanks for the detailed questions.

Just today I have fixed point 3 in our internal version of the codebase, what I did was add a check to Line 124 to only drop data if persistent_data_df actually contains data at that timestep. Further, I don't actually expect points 1 and 2 to work yet; I have not yet implemented the handling of unfrozen agents in simulation.

You understand the code correctly, and that is exactly why those errors come up. The way point 3 works is that only data from the agents existing in the initial frame of the scene is kept within persistent_data_df, this is why for unfrozen agents at later timesteps there is no corresponding data in the persistent_data_df as they did not exist in the initial frame when the simulation was created. Again, this is purely because I have not yet implemented the handling of unfrozen agents (it requires a deeper discussion overall, since agents appearing may not be consistent with how the scene evolved until then...).