Giters

automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Recreate a subset of benchmarks

svmhkk opened this issue · comments

commented

Hi, what would be the best way to recreate a subset of benchmarking accuracy data from the ZC json?
For ex: I would want to replicate the val accuracy obtained for the architecture 1-100 indexed from NASBench201. I understand that the val accuracies and corresponding ZC scores were open-sourced to remove the need for training, but I need to perform training for an internal evaluation. How do you suggest to go about this?
Thank you.

commented

Would you like to create a benchmark from NASBench201 Search Space's arches or the ZC scores? You can use Trainer in naslib/defaults to run and create benchmarks. If you want to create ZeroCost benchmarks, you can simply use ZeroCost available in predictors.

from naslib.predictors import ZeroCost
pred_type = "jacov"
zc_predictor = ZeroCost(method_type=zc_pred)
score = zc_predictor.query(graph=graph, dataloader=data_loader) 
# The graph is of type NASBench201 type in your case and dataloader would be a subset of CIFAR10 data.

Let me know, if that answers your issue.