Some errors about loading state_dict for FlowNetS

[Marco]

Hi, your works are so amazing. Thanks for your open source a lot.
There are two things that I want to seek help from you.

1. When I try to use the pre-trained model that you provide, but there are some errors. Then, I got a suggestion from the internet to fix them by using the code below.

model.load_state_dict(flownet_dict['state_dict'], False)

I wonder if that will cause any problems. And how can I load the model correctly?

PS: Here is my detailed code:

import torch
from models.FlowNetS import FlowNetS

model = FlowNetS()
model.eval()
flownet_dict = torch.load('/home/lab-huang.maochun/hmc/project/video-content-sentiment/frameLevel/myTransformer/flownets_EPE1.951.pth.tar')
model.load_state_dict(flownet_dict['state_dict'])

The error that I get

RuntimeError: Error(s) in loading state_dict for FlowNetS:
	Missing key(s) in state_dict: "conv1.1.weight", "conv1.1.bias", "conv1.1.running_mean", "conv1.1.running_var", "conv2.1.weight", "conv2.1.bias", "conv2.1.running_mean", "conv2.1.running_var", "conv3.1.weight", "conv3.1.bias", "conv3.1.running_mean", "conv3.1.running_var", "conv3_1.1.weight", "conv3_1.1.bias", "conv3_1.1.running_mean", "conv3_1.1.running_var", "conv4.1.weight", "conv4.1.bias", "conv4.1.running_mean", "conv4.1.running_var", "conv4_1.1.weight", "conv4_1.1.bias", "conv4_1.1.running_mean", "conv4_1.1.running_var", "conv5.1.weight", "conv5.1.bias", "conv5.1.running_mean", "conv5.1.running_var", "conv5_1.1.weight", "conv5_1.1.bias", "conv5_1.1.running_mean", "conv5_1.1.running_var", "conv6.1.weight", "conv6.1.bias", "conv6.1.running_mean", "conv6.1.running_var", "conv6_1.1.weight", "conv6_1.1.bias", "conv6_1.1.running_mean", "conv6_1.1.running_var". 
	Unexpected key(s) in state_dict: "conv1.0.bias", "conv2.0.bias", "conv3.0.bias", "conv3_1.0.bias", "conv4.0.bias", "conv4_1.0.bias", "conv5.0.bias", "conv5_1.0.bias", "conv6.0.bias", "conv6_1.0.bias". 

---

2. I want to use optical flow on my video dataset. So my video data size is [batch_size, frames, channel, weight, height]. So can I input my video data directly to get the optical flow? Here is my example:

artificial_video = torch.randn(1, 16, 3, 224, 224)
optical_flow = model(artificial_video)

[Clément Pinard]

1. You are trying to load a state dict of a model without batch norm into a flownet with batchnorm. This statedict was generated with anetwork using all default parameters. If you change the architecture by e.g. adding batchnorm it won't be compatible anymore. From my experience, batchnorm is really not that useful in this project, I kept the parameter for anyone to conduct further investigation, but if your sole goal is to have a functional optical flow network, you're better off with the default one.
2. You need to construct frame pairs. as such, you take to frames from your video frame1 = artificial_video[:, i] and frame1 = artificical_video[:, j], and then concatenante the channel dimension to get a [1, 6, 224, 224] tensor. input_tensor = torch.cat([frame1, frame2], dim=1) . It's important to note that the choice for the frame pair is arbitrary. The most obvious is to get consecutive frames, so j=i+1 , but you can totally take a larger gap between, should the video be not have large pixel displacement. Keep in mind that this network was trained with FlyingChairs, and thus will perform better with an optical flow that follows the same distribution, which means it needs to be pretty large, around 10px i'd say (but you can experimentally check this by doing some statistics over a subset of FlyingChairs optical flow maps)

[Marco]

Thanks for you detailed suggestion so much!