Prune and Replace NAS
by Kevin Alexander Laube and Andreas Zell, on arxiv
This code is based on the implementations of DARTS and P-DARTS.
Idea:
Iteratively pruning and replacing bad candidates from the operation pool enables us to efficiently search through vast operation spaces in reasonable time. We use DARTS to rank the currently available candidates, prune the worst ones, and generate new ones by applying network morphisms to those that are left.
Run:
Sample scripts that match our experiments in the paper are provided in the ./scripts folder.
Search
The progress and discovered cells of our DL2 search, these stats are plotted to tensorboard.
Retraining
Test error (%) on CIFAR-10 and CIFAR-100 in comparison. The focus of this research is the candidate operation space, thus the only major change compared to the DARTS baseline is the progressive pruning and replacement schedule.
| Method | #params | #ops | GPU days | CIFAR-10 | CIFAR-100 |
|---|---|---|---|---|---|
| NASNet-A | 3.3M | 13 | 1800 | 2.65 | |
| AmoebaNet-B | 2.8M | 19 | 3150 | 2.55 | |
| ENAS | 4.6M | 5 | 0.5 | 2.89 | |
| DARTS (1st order) | 2.9M | 8 | 1.5 | 2.94 | |
| DARTS (2nd order) | 3.4M | 8 | 4 | 2.83 | |
| P-DARTS C10 | 3.4M | 8 | 0.3 | 2.5 | 16.55 |
| P-DARTS C100 | 3.6M | 8 | 0.3 | 2.62 | 15.92 |
| sharpDARTS | 3.6M | 0.8 | 2.45 | ||
| SNAS moderate | 2.8M | 8 | 1.5 | 2.85 | |
| NASP | 3.3M | 7 | 0.2 | 2.8 | |
| NASP (more ops) | 7.4M | 12 | 0.3 | 2.5 | |
| PR-DARTS DL1 | 3.2M | 15/15 | 0.82 | 2.74 | 17.37 |
| PR-DARTS DL2 | 4.0M | 15/15 | 0.82 | 2.51 | 15.53 |
| PR-DARTS DR | 4.2M | 26/39 | 0.88 | 2.55 | 16.69 |
| PR-DARTS UR | 5.4M | 45/83 | 1.10 | 3.79 |
Citation
If you use any part of this code in your research, please cite our paper:
@article{laube2019prnas,
title={Prune and Replace NAS},
author={Kevin A. Laube and Andreas Zell},
journal={arXiv preprint https://arxiv.org/abs/1906.07528},
year={2019}
}