Temporal pooling.

[chandak]

Hi Tomas, could you please guide me on how I should implement the cross-channel weighted sum-pooling (temporal, basically) layer since I could not find anything on that ?

[Tomas Pfister]

Hi,

Sure, happy to help. Sorry, I haven't had the bandwidth to clean up the temporal code for release.

Given a video you'd want to:

1. Precompute dense optical flow
2. Modify the data layer so it reads in multiple frames and the optical flow between them
3. Add a warping layer which uses the optical flow to warp the heatmaps from the top spatial fusion layer (conv5_fusion in our prototxt)
4. Add a single 1x1 convolution on top of the warping layer output with an L2 loss to the ground truth (so you now have 3 losses) & give it a higher weight (say loss_weight: 5)
5. Train :)

We trained a single-frame network first (SpatialNet + Spatial Fusion Layers), initialised the above temporal network with its weights (plus gaussians for the added 1x1 convolution) & continued training.

Pull requests accepted if you or someone else has time to reimplement this in our public fork! :)

Hope that helps,
Tomas

[chandak]

Yes, that helps a lot Tomas. Thanks a lot. Will keep in touch in case I
have any further queries.

Sincerely.
On Mon, Apr 11, 2016 at 6:22 AM, Tomas Pfister [e-mail]
wrote:

Hi,

Sure, happy to help. Sorry, I haven't had the bandwidth to clean up the
temporal code for release.

Given a video you'd want to:

1. Precompute dense optical flow
2. Modify the data layer so it reads in multiple frames and the optical
   flow between them
3. Add a warping layer which uses the optical flow to warp the heatmaps
   from the top spatial fusion layer (conv5_fusion in our prototxt)
4. Add a single 1x1 convolution on top of the warping layer output with an
   L2 loss to the ground truth (so you now have 3 losses) & give it a higher
   weight (say loss_weight: 5)
5. Train :)

We trained a single-frame network first (SpatialNet + Spatial Fusion
Layers), initialised the above temporal network with its weights (plus
gaussians for the added 1x1 convolution) & continued training.

Hope that helps!
Tomas

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://contactmonkey.com/api/v1/tracker?cm_session=d859a7fc-e9e1-4cb3-b843-91b6b27560cc&cm_type=link&cm_link=ce95b597-c770-40fb-9b30-5637b005eaa3&cm_destination=https://github.com/tpfister/caffe-heatmap/issues/9#issuecomment-208103982

Regards,
Chinmay Mahesh Chandak.
3rd year undergraduate,
Computer Science and Engineering,
IIT Hyderabad.

[zimenglan]

hi @tpfister, can you release the result of SpatialNet + Spatial Fusion Layers on flic dataset?

[Yixin Jin]

Hi @tpfister how do you use the optical flow to warp 64x64 heatmaps? Thank you very much

[Apes]

Hi, @jinyixin621
Have you found the answer yet?

Hi @tpfister how do you use the optical flow to warp 64x64 heatmaps? Thank you very much

I do not know how they warp optical flow file which is use deepflow2 precomputed from dataset to heatmaps like they said in paper ,have this part code released?

[Yixin Jin]

Hi, @mindcont I didn't get reply from the author, but I implement one method by my own understanding.

[Abolfazl kaviani]

Hi @jinyixin621 , where can I find your implementation on warping heatmap layer?

Hi, @mindcont I didn't get reply from the author, but I implement one method by my own understanding.