NASBench201 KeyError

[YangZhaohui]

Thanks for sharing the great repo. I tried to access the accuracy of 'x-valid' on CIFAR10 dataset but failed because of the key error.

>>> results = api.query_by_index(0, 'cifar10')
>>> results.keys()
dict_keys([777, 888, 999])
>>> results[888].get_eval('x-valid')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yangzhaohui/anaconda3/lib/python3.7/site-packages/nas_201_api/api.py", line 885, in get_eval
    'loss'    : self.eval_losses['{:}@{:}'.format(name,iepoch)],
KeyError: 'x-valid@199'

[Xuanyi Dong]

I will have a look at this problem today and get back to you soon.

[YangZhaohui]

ok, waiting for you.

[Xuanyi Dong]

Thanks for your patience, to access the validation information, please use:

results = api.query_by_index(0, 'cifar10-valid')
results[888].get_eval('x-valid')

[YangZhaohui]

Problem solved, thank you!

train -> results[key].get_train()['accuracy']
cifar10-valid -> results[key].get_eval('x-valid'/'ori-test')['accuracy']

[Xuanyi Dong]

You are welcome! Thanks for using our benchmark :)

[YangZhaohui]

It's a great work and everyone should use the benchmark to verify their search algorithm!