What is the network architecture corresponding to "sss" search space?
hengzhe-zhang opened this issue · comments
Which Algorithm
NATS-Bench
Describe the Question
I have read the documentation, in which mentioned the training codes defined in "main-sss.py". However, in this file, I cannot find any information about the specific network architecture. Therefore, it makes me confused about the meaning of the search space, i.e., what is the concrete meaning of those five channel sizes. Besides that, I find that there is a function that seems extremely important since it may contain the definition of the search space, i.e. "bench_evaluate_for_seed". However, I cannot find the corresponding codes related to this function. In summary, I feel so confused about the definition of the search space. I will be grateful if you can answer my question.
@zhenlingcn, thanks for your interest.
The architecture is |nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|nor_conv_3x3~1|nor_conv_3x3~2| mentioned in https://github.com/D-X-Y/AutoDL-Projects/blob/main/exps/NATS-Bench/main-sss.py#L12, which is the best architecture in the topology search space on CIFAR-100.
bench_evaluate_for_seed is a function to train-and-evaluate a single network with a given seed and optimization config (https://github.com/D-X-Y/AutoDL-Projects/blob/main/xautodl/procedures/funcs_nasbench.py#L83)
Okay, thanks for your reply! However, what's the meaning of architecture "|nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|nor_conv_3x3~1|nor_conv_3x3~2|"? Is there any interpretation documentation? I need to leverage this benchmark in my work, so I hope I can draw a picture to demonstrate the full search space rather than just a schematic diagram illustrated in your paper.
|nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|nor_conv_3x3~1|nor_conv_3x3~2| means:
- The first node's output is a 3x3-convolution applied to (input-of-the-this-cell). <->
|nor_conv_3x3~0| - The second node's output is 3x3-convolution(input-of-the-this-cell) + 3x3-convolution(output-of-the-first-node).
- The third node's output as well as this cell's output is input-of-the-this-cell + 3x3-convolution(output-of-the-first-node) + 3x3-convolution(output-of-the-second-node)
This is also visualized in the top-row of Figure-1 in https://arxiv.org/pdf/2009.00437.pdf.
Great, do you mean these five parameters are just used to set the channel size of these five convolution layers?
No. The five parameters in sss is used to set the number of channels for different cells and residual layers.
The five conv layers within a cell has the same number of channels.
Okay, thanks for your answer. I understand the search space now.