Giters

D-X-Y / AutoDL-Projects

Automated deep learning algorithms implemented in PyTorch.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Reproduce NAS-Bench-201 REA algo

JackFram opened this issue · comments

commented

Appreciated for your great work!

Which Algorithm
REA for NAS-Bench-201

Describe the Question
when running bash ./scripts-search/NAS-Bench-201-algos/BOHB.sh cifar10 -1
the error

Traceback (most recent call last):
  File "./exps/NAS-Bench-201-algos/reinforce.py", line 290, in <module>
    save_dir, index = main(args, nas_bench)
  File "./exps/NAS-Bench-201-algos/reinforce.py", line 187, in main
    reward, cost_time = train_and_eval(arch, nas_bench, extra_info, dataname)
  File "/home/ubuntu/zzh/XAutoDL/exps/NAS-Bench-201-algos/R_EA.py", line 111, in train_and_eval
    raise ValueError("NOT IMPLEMENT YET")
ValueError: NOT IMPLEMENT YET

occurred, any workaround?

commented

It is because that you did not download the benchmark file. See here, you need to download NAS-Bench-201-v1_1-096897.pth and put it into ${TORCH_HOME}.

In addition, the NAS-Bench-201 has been upgraded to NATS-Bench, and it is suggested to use the BOHB script in NATS-Bench: https://github.com/D-X-Y/AutoDL-Projects/blob/main/exps/NATS-algos/bohb.py#L8
In short, three key difference for your target experiment are:

  • NATS-Bench contains more results of each architecture candidate
  • Optimized API -- faster and more robust
  • Better tuned hyperparameters for BOHB as well as other baselines.
commented

Cool, my mistake, I ran REA instead of BOHB and there is a line in the code indicating not implemented yet. Anyway, during the traceback, it seems still to be a nas bench API location problem. Appreciated!

commented

No problem~ I should print more informative debug info.