Why ENAS gets a poor accuracy on NAS-Bench-201?
xzhou29 opened this issue · comments
Xin commented
Describe the bug
Why it would have a much higher accuracy with just skip connections?
To Reproduce
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/ENAS.sh cifar10 1 -1
Ran twice with these two seed numbers (randomly generated): 49363, 65491
Expected behavior
The results just do not make any sense to me.
Screenshots
seed-49363:
Loss=1.238, Accuracy@1=56.10%, Accuracy@5=95.49%
ENAS : run 250 epochs, cost 7187.7 s, last-geno is Structure(4 nodes with |skip_connect~0|+|skip_connect~0|skip_connect~1|+|skip_connect~0|skip_connect~1|skip_connect~2|).
seed-65491:
Loss=3.963, Accuracy@1=10.88%, Accuracy@5=66.97%
ENAS : run 250 epochs, cost 7059.1 s, last-geno is Structure(4 nodes with |avg_pool_3x3~0|+|skip_connect~0|avg_pool_3x3~1|+|skip_connect~0|avg_pool_3x3~1|skip_connect~2|).
Xuanyi Dong commented
This is what we have discussed in our paper. ENAS and DARTS will quickly converge to the models that can converge fast but has low accuracy.
Xin commented
I didn't notice that on the paper.
Thank you so much!
Xuanyi Dong commented
No problem, please let me know if you have any other questions.