Giters

NVlabs / trajdata

A unified interface to many trajectory forecasting datasets.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Integrating trajdata into trajectron++

Leon0402 opened this issue · comments

commented

Hi,

currently I'm trying to integrate trajdata into T++. I started with the simplest approach of not touching T++ code at all and basically rewrite the dataset specific preprocessing code such as this and use trajdata for it.
While this works (at least with the simple eth dataset without maps, I haven't implemented nuscenes yet) I don't like the approach very much as it is basically only converting from one data format to the other and none of the features of trajdata are really used.

Instead I would like to integrate trajdata more deeply into T++ using it directly with pytorch Dataloader. This is probably what you're doing as well.

My thoughts on how to this would be the following:

Hope this makes sense. There are a few issues though:

Thanks for your help!

It would be super helpful to see your work in progress implementation of trajdata in Trajectron. This would probably resolve my questions and speed up my development. If you company rules allow you to share this, I would be very grateful.
(Uploaded as a git patch here for example)

commented

Hi @Leon0402, my apologies for the delay in responding (I'll also make sure to respond to your other issues soon)!

Indeed, I plan to release an updated Trajectron++ codebase that uses the dataloader. Unfortunately, I'll have to wait until after my next paper deadline since I'm using T++ with the dataloader in that work and will have to go through the proper IP review channels before releasing it (as you guessed).

Having said that, everything you said above is correct. Let me answer the questions you have outstanding:

The previous code of NodeTypeDataset multiplies some scenes and nodes. Is this still necessary with trajdata? How would I do this?

Good question, I personally just did away with that multiplying (it was an attempt to fix the crazy imbalance in the data, where straight-line driving dominates) and found that the model still performs fine. Admittedly, I don't have concrete evaluation numbers just yet, but I'll get them soon and then we'll see if I'm totally wrong about that or not.

More broadly, I haven't implemented a dataset sample weighting scheme yet (likely I would just use the weighted sampling schemes offered by PyTorch anyways), but maybe PyTorch's own sample weighting capabilities would be a good place to start looking if you want to have that for your model.

How do I work with the temporal scene graph with trajdata? T++ uses such a graph for the edge encoding (https://github.com/StanfordASL/Trajectron-plus-plus/blob/58b9763d02b9c90e1b67dad1026bf0b1b9e322a9/trajectron/model/dataset/preprocessing.py#L115). Trajdata offers something similar with agent_interaction_distances, which doesn't support hyperparameters like edge_addition_filter and edge_removal_filter though.

I completely don't use the temporal scene graph anymore, instead I directly use the hyperparams['edge_addition_filter'] in the edge encoder in mgcvae.py