f4nku4n / evoxbench

A neural architecture search benchmark without requirements of GPUs or PyTorch/TensorFlow

Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment [arXiv]

Introduction to EvoXBench

• Click on the image to watch the video.

• Please note that the calculation of IGD is only applicable to problems derived from search spaces that can be exhaustively evaluated (i.e., C-10/MOP1 - C-10/MOP7). This is because the true Pareto Fronts are available for such problems. For problems derived on the basis of surrogate models (i.e., C-10/MOP8 - C-10/MOP9 and IN-1K/MOP1 - IN-1K/MOP9), the true Pareto Fronts are unknown and we cannot calculate IGD.

Preparation Steps

1. Download the following two requried files:

   • database.zip file from Google Drive or Baidu云盘（提取码：mhgs）

   • data.zip file from Google Drive

     or Baidu云盘（提取码：lfib）

2. pip install evoxbench to install the benchmark.

3. Configure the benchmark via the following steps:

    from evoxbench.database.init import config

    config("Path to databae", "Path to data")
    # For example
    # If you have the following structure
    # /home/Downloads/
    # └─ database/
    # |  |  __init__.py
    # |  |  db.sqlite3
    # |  |  ...
    # |  
    # └─ data/
    #    └─ darts/
    #    └─ mnv3/
    #    └─ ...
    # Then you should do:
    # config("/home/Downloads/database", "/home/Downloads/data")

Database

Visit this webpage for more information: https://github.com/liuxukun2000/evoxdatabase

Support

• You can ask any question in issues block and upload your contribution by pulling request (PR).
• If you have any question, please join the QQ group to ask questions (Group number: 297969717).

Acknowledgement

Codes are developed upon: NAS-Bench-101 , NAS-Bench-201, NAS-Bench-301 , NATS-Bench , Once for All , AutoFormer, Django , pymoo