Rotation augmentation seems incorrect

[Mickaël Seznec]

Hi,

In the data augmentation code, the rotation map is computed directly from the coordinates of the flow:

FlowNetPytorch/flow_transforms.py

Lines 188 to 192 in 79894dc

 def rotate_flow(i,j,k): 
     return -k*(j-w/2)*(diff*np.pi/180) + (1-k)*(i-h/2)*(diff*np.pi/180) 
  
 rotate_flow_map = np.fromfunction(rotate_flow, target.shape) 
 target += rotate_flow_map 

Instead, I think it should rather rotate the flow according to where it "lands", with something like this:

# 1. Compute the warped coordinates
grid = np.mgrid[:h, :w].transpose(1, 2, 0) # generate the coordinate grid
warped_grid = grid + np.flip(flow, axis=2) # must flip flow to respect the [y, x] ordering


# 2. Apply the rotation on the warped coordinates
# Note that I'm not using a small angle approximation here, but it could perfectly be possible
flow_rotation_matrix = np.array([
            [np.sin(diff * (np.pi / 180))    ,  np.cos(diff * (np.pi / 180)) - 1],
            [np.cos(diff * (np.pi / 180)) - 1, -np.sin(diff * (np.pi / 180))    ]
])
warped_grid_rotated = np.apply_along_axis(
       lambda coord_pair: flow_rotation_matrix @ (coord_pair - np.array([h / 2, w / 2])),
       2,
       warped_grid
)

flow += warped_grid_rotated

I'm not 100% confident in this, have you already thought about that?

Thanks,
Mickaël

[Clément Pinard]

Hi,

I think you are right.
When e.g. we take the point in the center of rotation, and suppose it moved outside of this center of rotation, the added optical flow to this point is not zero.

As such, my implementation, albeit very simple is only possible when assuming small angles AND small optical flow.

We discussed about a more rigourous implementation of optical flow augmenation here : NVIDIA/flownet2-pytorch#26 . For simplicity I was keeping my implementation, but that was only considering the small angles assumption. I don't think we can make the small optical flow assumption, especially with Flying chairs.

How fast is your implementation ? If it doesn't make the data augmentation part the computation bottle neck, I'd happily merge a pull request.

Anyway, thanks a lot, good catch !