automl / NASLib

Hi, thank you for sharing these benchmark tools!

Let me ask one question about a dataset provided by this repo.

When I looked at data in NAS-Bench-301 trained on CIFAR-10 whose URL is provided in README.md in this repo, the format of architecture is supposed to be encoded numerically unlike NAS-Bench-201 (see also the code and its output below).

import pickle

data = pickle.load(open("./nb301_full_training.pickle", "rb"))
print(list(data.keys())[0])

(((0, 6), (1, 4), (0, 0), (1, 5), (1, 4), (3, 2), (0, 6), (3, 2)),
 ((0, 1), (1, 4), (0, 1), (1, 4), (2, 6), (3, 4), (1, 5), (2, 1)))

I'm wondering if we could know the encoding rule somewhere. I tried to do so but I found it difficult because we have at least four related NAS-Bench-301 repositories as follows.

https://github.com/crwhite14/nasbench301: this repo is mentioned by README.md in https://github.com/automl/NASLib to use NAS-Bench-301.
https://github.com/automl/nas-bench-x11: The forked version refers to this repo as the full code is now available here: https://github.com/automl/nas-bench-x11..
https://github.com/automl/nasbench301: This looks newer than https://github.com/crwhite14/nasbench301.
this repo

Best regards,

hi @Neonkraft, could you kindly take a look at my question as an author of #106?

Hi @nzw0301,

Thank you for your interest in our project and sorry for the delay in replying. The conversion to the NASLib graph can be found here:

NASLib/naslib/search_spaces/darts/conversions.py

Line 310 in 1dd5507

def convert_compact_to_naslib(compact, naslib_object):

I hope this helps.

Hi @Neonkraft, I really appreciate for answering my question. I'll definitely check the conversion part. If I have some problem, let me ask a follow-up question here. Otherwise, I'll close this PR.

Hi, can I ask how to create naslib_object?

According to convert_genotype_to_naslib, it might be empty DARTSSearchSpace() object. However, when I ran the following code,

from naslib.search_spaces import darts
naslib_object = darts.graph.DartsSearchSpace()

I could not instantiate the instance correctly due to the following error.

AttributeError                            Traceback (most recent call last)
Input In [10], in <cell line: 1>()
----> 1 darts.graph.DartsSearchSpace()

File ~/Documents/NASLib/naslib/search_spaces/darts/graph.py:70, in DartsSearchSpace.__init__(self)
     62 def __init__(self):
     63     """
     64     Initialize a new instance of the DARTS search space.
     65     Note:
   (...)
     68         before initializing the class. Default is 10 as for cifar-10.
     69     """
---> 70     super().__init__()
     72     self.channels = [16, 32, 64]
     73     self.compact = None

File ~/Documents/NASLib/naslib/search_spaces/core/graph.py:106, in Graph.__init__(self, name, scope)
     86 """
     87 Initialise a graph. The edges are automatically filled with an EdgeData object
     88 which defines the default operation as Identity. The default combination operation
   (...)
    103 
    104 """
    105 # super().__init__()
--> 106 nx.DiGraph.__init__(self)
    107 torch.nn.Module.__init__(self)
    109 # Make DiGraph a member and not inherit. This is because when inheriting from
    110 # `Graph` note that `__init__()` cannot take any parameters. This is due to
    111 # the way how networkx is implemented, i.e. graphs are reconstructed internally
   (...)
    121 
    122 # self._nxgraph.edge_attr_dict_factory = lambda: EdgeData()

File /opt/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/networkx/classes/digraph.py:319, in DiGraph.__init__(self, incoming_graph_data, **attr)
    317 # clear cached adjacency properties
    318 if hasattr(self, "adj"):
--> 319     delattr(self, "adj")
    320 if hasattr(self, "pred"):
    321     delattr(self, "pred")

File /opt/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/torch/nn/modules/module.py:1181, in Module.__delattr__(self, name)
   1180 def __delattr__(self, name):
-> 1181     if name in self._parameters:
   1182         del self._parameters[name]
   1183     elif name in self._buffers:

File /opt/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module.__getattr__(self, name)
   1128     if name in modules:
   1129         return modules[name]
-> 1130 raise AttributeError("'{}' object has no attribute '{}'".format(
   1131     type(self).__name__, name))

AttributeError: 'DartsSearchSpace' object has no attribute '_parameters'

Note that I install dependencies by calling pip install -r requirements.txt.

Hi @nzw0301, it looks like this is due to an update in the latest version of networkx. Could you uninstall it and install version 2.8? I've also updated requirements.txt.

You can use the following snippet to get the model:

compact = (((0, 6), (1, 4), (0, 0), (1, 5), (1, 4), (3, 2), (0, 6), (3, 2)), ((0, 1), (1, 4), (0, 1), (1, 4), (2, 6), (3, 4), (1, 5), (2, 1)))
graph = DartsSearchSpace()
graph.set_spec(compact)
graph.prepare_evaluation() # Skip this step if you don't want the full model (more cells stacked, more channels in convolutions)
graph.parse()
output = graph(torch.randn(2, 32, 3, 3))

@Neonkraft Brilliant! Thanks!

How to recover the search space from `NAS-Bench-301`?