hello, i am curious about whether do you have not use any activate functions in the generator network?

[chensongkui]

[Xintao]

@chensongkui
Activation functions are used in the generator network. You can see the network structure in pytorch_test/architectures.py

[chensongkui]

Sorry, I don't understand very well in the code. Whether each layer of the convolutional layer on the generator architecture is followed by an activation layer？

[chensongkui]

because the generator architecture showed on your paper don't include any activation layer

[Xintao]

@chensongkui
Some of the convolutional layers are followed by an activation layer but not all.
We did not put all the details of the generator in the paper.

[chensongkui]

Thank you!

[chensongkui]

hi, i am curious about whether your training dataset from ImageNet is first pre-processed into low-resolution dataset part and high-resolution dataset part before training your generator network?

[KewJieLong]

@chensongkui

Detail about generator can be found on Supplementary Material (http://mmlab.ie.cuhk.edu.hk/projects/SFTGAN/suport/cvpr18_sftgan_supp.pdf)
The icon stated which activation is used (Relu and leaky Relu activation).

PyTorch Code for SFT layer can be found at https://github.com/xinntao/BasicSR/blob/master/codes/models/modules/sft_arch.py#L40
From what i observed, there are only 3 layer without activation:

1. First layer (do not use activation so that able to do summation with the output of last layer?)
2. Layer after 16 block of ResBlock (do not use activation so that able to do summation with the output of first layer?)
3. Last layer (Inferencing a image which activation function is not needed?)

However, this is just my guess of the why the architecture design in such a way. @xinntao Please correct me if there is any thing wrong. Thank you

[chensongkui]

Thank you! @KewJieLong

[Xintao]

Thanks @KewJieLong
@chensongkui

For the activation in the model, there are no clear and strict rules and the influence is minor.
I usually consider the followings for model design: 1) the common practice in SR; 2) other model designs in high-level vision tasks, for example, Identity Mappings in Deep Residual Networks from Kaiming.

Let me give more details.
We can roughly divide the conv into 3 parts. 1) conv in residual blocks in LR size; 2) conv in the main path in LR size; 3) conv in the main path in HR size. Here the opposite of the main path is the residual paths.

1. conv in residual blocks in LR size
   SRGAN: ignoring BN layes, Conv-PReLU-Conv;
   EDSR: Conv-ReLU-Conv;
   Identity Mappings in Deep Residual Networks recommends using pre-activation residual blocks, i.e., ReLU-Conv-ReLU-Conv.

So, in our torch version, we use ReLU-Conv-ReLU-Conv type and in our pytorch version, we use Conv-ReLU-Conv. It does not influence much.

2. conv in the main path in LR size
   The main idea from Identity Mappings in Deep Residual Networks is that we should keep a “clean” information main path (as shown in Figure 1 in their paper) and do not put activation layer in the main path. So we do not put ReLU after the first conv layer and the layer after 16th ResBlock.

3. conv in the main path in HR size
   But I still use the activation layer in HR main path since there is no residual block in HR main path.

[jiequanz]

A simple question. Are torch and pytorch two different things? I thought they were the same....

[Xintao]

Different languages: Torch is in Lua, PyTorch is in python.
But they have the same core lib. PyTorch is developed based on Torch.