Min history & future length not working for SimulationScene

[Leon De Andrade]

The configuration parameters history_sec and future_sec (the min part to be more specific) don't work with SimulationScene. The Scene returned by get_obs still includes agent, which have a shorter history / future.

According to some comment, this isn't explicitly checked, because dataset already took care of it:
https://github.com/nvr-avg/trajdata/blob/24251636bc708bd20e91a538b68095300a3199c8/src/trajdata/caching/df_cache.py#L224

I stepped through dataset and saw that everything there worked as expected in agent mode at least (dataset._get_data_index_agent). I haven't tested it, but I suppose it might be not working in scene mode as the code for this (dataset. _get_data_index_scene) only removes complete scenes if no agent matches the constraint. So a scene with at least one matching agent will still contain all the other agents, which don't match the constraint? Or am I missing something here?

The above mentioned methods are called once in the constructor (only length) and in the get mode, so it works as expected with DataLoader. But I'm not quite sure how SimulationScene uses this information or if it uses the information all.

[Boris Ivanovic]

I haven't tested it, but I suppose it might be not working in scene mode as the code for this (dataset. _get_data_index_scene) only removes complete scenes if no agent matches the constraint. So a scene with at least one matching agent will still contain all the other agents, which don't match the constraint

This is exactly correct, and it is intended. I wanted to err on the side of data inclusion, since the other way around (requiring every single agent in a scene to have the minimum amount of data) may lead to almost zero data being returned to the user. It's a bit of an open question as to what's the best way to handle this, but the current way will at least return a good amount of data (where at least one agent satisfies all the requirements).

But I'm not quite sure how SimulationScene uses this information or if it uses the information all.

It's used in the get_obs function, take a look here.

[Leon De Andrade]

This is exactly correct, and it is intended. I wanted to err on the side of data inclusion, since the other way around (requiring every single agent in a scene to have the minimum amount of data) may lead to almost zero data being returned to the user. It's a bit of an open question as to what's the best way to handle this, but the current way will at least return a good amount of data (where at least one agent satisfies all the requirements).

But isn't the purpose of min history / future to have a hard requirement. Say I have a model , which requires at least x steps in the history to do the prediction: Then it's problematic if the data still contains agents, which have less steps in the history.
So my assumption was that scene centric mode would give me all scenes back, where at least one agent satisfies the condition (as it is now), but additionally removes agents from these scenes that don't satisfies the condition.

Example Scene 0 has 4 agents and half of these satisfies the condition. The dataloader would give me that scene (because at least one agent satisfies the condition), but the scene data would only contain the data of two matching agents and not of the other two.

I'm not quire sure if this makes sense to be honest (missing the experience here) :-) Can you perhaps clarify when one should use scene centric mode? Perhaps I'm using it wrong right now. Currently I use it for prediction.

It's used in the get_obs function, take a look here.

Ah sorry I should clarify.

• So this initializes AgentBatch with history_sec / future_sec
• The constructor then uses this information to fetch the history here and similary for the future
• The cache will return the agent history here

But the cache won't filter for minimum history. It says in the comment:

We don't have to check the mins here because our data_index filtering in dataset.py already took care of it.

And that is the part I currently don't understand. Because I think the data_index filtering didn't take care of it.

But maybe it's the same misunderstanding as with scene mode. Currently I use SimulationScene for evaluation. So with sim_scene.step(...) I get the current AgentBatch. And that AgentBatch contains agents that don't satisfy my min history / future length. This is problematic because my ground truth data my contain less elements than my prediction in this case.
So I have to manually filter all agents that don't satisfy the condition.

Very easy to reproduce, just use some dataset like eth and initalialize it with init_timestep=0. At this point, no agent has any history. But sim_scene.get_obs()will still return anAgentBatchwith all agents in it. So no filtering was done here. My expectation would be thatSimulationScenethrows aValueError`, because the scene is empty after filtering.

[Boris Ivanovic]

Sorry for the delayed response @Leon0402 !

I'm not quire sure if this makes sense to be honest (missing the experience here) :-) Can you perhaps clarify when one should use scene centric mode? Perhaps I'm using it wrong right now. Currently I use it for prediction.

There are some problems with just removing agents from a scene, chiefly that those removed agents will have had some effect on the remaining agents in the scene (like, in the real-world which is then reflected in the data). This is problematic from a modeling perspective because it makes it very difficult for your model to reason about why an agent moved in a certain way, if the reason why is that there was some other agent blocking its path (which was removed because it didn't satisfy the history requirement or something like that).

This also ties in to your second question. Scene-centric refers to a model that predicts the futures of all agents in a scene simultaneously. In contrast, an agent-centric model predicts the future of one agent at a time (taking into account other agents, but such a model's predictions are independent per agent). To give concrete examples, Trajectron++ is an agent-centric model, and ScePT is a scene-centric model.