Datasets in the benchmark with download links:
- CIFAR-100 (Image classification)
- Spherical CIFAR-100 (Transformed image classification) (272 MB)
- Ninapro DB5 (Hand-gesture classification)(15 MB)
- Darcy Flow (Partial differential equation solver) (1.6 GB)
- PSICOV (Protein sequence distance prediction) (1.1 GB)
- FSD50k (Sound event classification) (24 GB)
- Cosmic (Cosmic ray identification and replacement) (6.5 GB)
- ECG (Cardiac anomaly detection)(150 MB)
- Satellite (Earth monitoring through satellite imagery) (322 MB)
- DeepSEA (identifying chromatin features from DNA sequences)(860 MB)
Precomputed evaluation benchmark files on the NB201 search space (following NATS-Bench):
- NinaPro DB5(84 MB)
- Darcy Flow (85 MB)
Full outputs (include training logs):
- NinaPro DB5(46 GB)
- Darcy Flow (35.4 GB)
We use the open-source Determined software to implement experiment code.
Installing determined: pip install determined
A master instance is required:
-
for local deployment (need to install docker):
- to start the master:
det deploy local cluster-up
- access the WebUI at
http://localhost:8080
- to shut down:
det deploy local cluster-down
- to start the master:
-
for AWS deployment (preferred):
- install AWS CLI
- Run
aws configure
and find AWS EC2 keypair name - to start the master:
det deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR_NAME
- access the WebUI at
{ec2-instance-uri}:8080
- to shut down:
det deploy aws down --cluster-id CLUSTER_ID
For an end-to-end example of running experiments with determined, you can refer to this video.
When running experiments, a docker image is automatically pulled from docker hub which contains all required python packages , i.e. you don't need to install them yourself, and it ensures reproducibility.
We provide pytorch implementations for two state-of-the-art NAS algorithms: GAEA PC-DARTS (paper link) and DenseNAS (paper link), which can be found inside each folder with the associated name, i.e. "darts/" for GAEA PC-DARTS and "densenas/" for DenseNAS.
To run these algorithms on 1D tasks, we've adapted their search spaces whose experiments are provided in "darts_1d/" for GAEA PC-DARTS (1D) and "densenas_1d/" for DenseNAS(1D).
Two task-specific NAS methods are implemented: Auto-DeepLab for dense prediction tasks in "autodeeplab/" and AMBER for 1D prediction tasks in "AMBER/".
We also implement procedure for running and tuning hyperparameters of the backbone architecture Wide ResNet (paper link), in "backbone/". The 1D-customized Wide ResNet is in "backbone_1d/".
To modify the random seed for each experiment, modify the number under
reproducibility: experiment_seed:
for each script