Intermediate output before the refinement part

[Akshay Iyer]

Hello,

Could anyone throw any light on what could the intermediate representation of the network be just after the Conv layers before the refinement network?
Finally an optical flow image is produced but what could a possible output be after the Conv part for a given pair of images as input?

The reason I'm asking is that I've been reading a few papers on achieving visual odometry using deep learning, all of which use the first part of the FlowNetS model without the refinement part. This output is then typically fed into an LSTM.

Any help would be greatly appreciated.

Cheers,
Akshay

[Clément Pinard]

You won't get any straight answer for this question, other than "we don't know, it just works"

The key here is that the last feature contains everything needed to reconstruct an optical flow estimation, since it's the last layer.

As such, any task for which the optical flow could be sufficient might use this feature map as a source for "abstract information" which serves as a basis for optical flow reconstruction AND odometry.

But while reading my answer you could be tempted to say "why not use optical flow map ? Why not use raw images ? These are also "feature maps" where every information needed for optical flow is available ? Hence the heuristic part, author could have used any kind of feature map throughout the whole network, it just happens that the last layer works better. You apprently need a lot of spatial information, hence the last layer, but at least a little bit of semantic information, hence not using direct optical flow.