Original Bias is not added with shifting parameter?

[Jansen Domgouen]

Hello, I hope to clarify since the original bias term of a convolutional layer seems to be missing in the implementation? In your paper specifically in Equation 6 and Figure 3. the original bias term "b" is shifted by the shifting parameter \Phi_{S_{2}} as part of the scale and shift (SS) parameters.

I noticed that in the function "block_forward" found in the script ./tensorflow/models/resnet12.py, particularly the line:
net = resnet_conv_block(inp, self.process_ss_weights(weights, ss_weights, block + '_conv1'),
ss_weights[block + '_bias1'], reuse, scope+block+'0')

I understand that the first three arguments of "resnet_conv_block" function (found in ./tensorflow/utils/misc.py) takes "inp" input, "cweight" that is the filter's weights, "bweight" which is the biases for this conv layer.
It seems that cweight is only the 'scaled' version of the convolution filters EXCLUDING the bias as returned by the function "process_ss_weights".
However in "resnet_conv_block" the conv_output is the convolved input with the filter cweight +bweight. However the bweight argument is only the ss_weights[block + '_bias1']. It seems it doesn't follow the Eq.6 in the paper which should add the weights[block+'_bias(1,2,3)']+ss_weights[block + '_bias1'].

So does this mean that we dispose of the original bias term of the pretrained resnet-12?

Thank you for your time.

[Yaoyao Liu]

Thanks for your interest in our work.

It is true that the shifted bias is not applied in the open-source implementation. We conducted extensive experiments on different datasets and observed that the bias doesn't have a very strong influence on the final performance. In some project, the bias is even not applied, e.g., FEAT:
https://github.com/Sha-Lab/FEAT/blob/e226a0466bb4366769426f2a519791aea16394d3/feat/networks/resnet.py#L4

To make the open-source code simple and clean, the shifted bias is removed. It would be very easy to add the shifted bias back. You could run the experiments yourself to see the performance with and without shifted bias.

If you have any further questions, feel free to add additional comments on the issue.

[Jansen Domgouen]

That explains it. Thank you very much for your time!