Giters

automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Extra .json files for zc_intro.py

AnaRisnoveanu opened this issue · comments

commented

I am running tutorial/zc_intro.py and it returns this error:
`[Errno 2] No such file or directory: '/home.../naslib/data/zc_nasbench201.json'

File "/home.../naslib/utils/get_dataset_api.py", line 24, in get_zc_benchmark_api
with open(datafile_path) as f:
File "/home.../tutorial/zc_intro.py", line 113, in
zc_api = get_zc_benchmark_api('nasbench201', 'cifar10')

FileNotFoundError: [Errno 2] No such file or directory: '/home.../naslib/data/zc_nasbench201.json'`

I have already checked the data for the benchmark but this file was not there to download.

commented

Hi @AnaRisnoveanu,

You could run the following bash script to download the missing json file:

bash scripts/zc/bash_scripts/download_nbs_zero.sh nb201

You could also pass the arguments nb101, nb301, and tnb101 to this bash script for the respective search spaces.

Alternatively, you can also download using these drive links:
NB101
NB201
NB301
TNB101-Macro
TNB101-Micro

We will soon update the readme about it.

commented

Thank you, zc_intro.py works fine now NAS-Benchmark-201.

When I set it to NAS-Benchmark-101, I get an error in score = zc_api[str(spec)][pred]['score'] on line 118. There is no str(spec) in the keys of zc_api.

Exception has occurred: KeyError '(0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 2, 3, 3, 1)' File "/home/anamariar/topush/experimental.nas_ebs/tutorial/zc_intro.py", line 118, in <module> score = zc_api[str(spec)][pred]['score'] KeyError: '(0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 2, 3, 3, 1)'

commented

There are 423624 total architectures available in NASBench101 with CIFAR10, but we only have a list of around 10000 in the NASBench101 zero-cost benchmark. That's the reason you are getting this error.

The search spaces having an exhaustive list of the zero-cost benchmarks are only NasBench201, TNB101 Micro, and TNB101 Macro.

You may also refer to the NAS-Bench-Suite-Zero paper for exact numbers and read more about the reports regarding using zero-cost proxies with NAS.