PIEPredict vs. SF-GRU

[stratomaster31]

Thank you for your fantastic work. I'm really enjoying your papers:

• PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction
• Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs

These two approaches solve different problems: trajectory prediction and action anticipation. I have some concerns/ideas I'd like to share:

1. Intention Prediction/Classification shows a F1 score higher than Action Classification in your experiments (0.87 vs. 0.72). It's quite a big difference. Do you think is a matter of the labels balance for both cases?
2. Do you think Action Classification could also benefit from previous Intention Classification?
3. Intention Classification is only present in your dataset, so training your models over other datasets is not possible. Do you think that trajectory prediction could also be refined with Action Classification? I mean, replacing the Intention Classification model by an Action Classification model
4. I noticed in your models that you first train the intention prediction model and then, the trajectory predicition one. Could it be interesting to set some common parts to both networks to train a multi-task learning fashion model (both models at the same time)?

Thank you very much again for your work, it is very inspiring!

[Amir Rasouli]

Hi,

Thanks for your comments. I think all the points you have made are very valid and worth trying. To answer your questions:
1- That certainly is one possibility. The data is not balanced and the network does favor the more represented class.
2- I believe it does. We have done some experimentations on that in one of our upcoming papers and saw its benefit in action classification
3- Regarding the dataset, training of the intention model can always be done on PIE and then just be used as part of the network on any other datasets. Regarding action for trajectory, Yes can definitely benefit trajectory prediction. This was highlighted in some recent publications at CVPR in the context of surveillance. The authors showed that activity recognition can improve trajectory prediction.
4- I totally agree with you. Intention in the framework we presented in our paper can be learned alongside trajectories. One reason why we did it in the way presented in the paper is that the intention data should resemble the conditions in which we ran our human experiments for the ground truth data to be valid. For trajectory prediction, however, we do not have such restrictions, therefore we can train it on many more samples. In this context, one could fine-tune the intention network at the end as part of the final model. I also agree with you, intention can also be formulated as a separate task beside trajectory instead of an input to the network, i.e. formulate it into a multi-task learning problem. I believe many variations of this should be examined. We only presented one of many possibilities.

[stratomaster31]

Hi again,

Thank you for your feedback! I'll follow your work closely :)