The performance analyzer for Iterative Optimization Heuristics (IOHs).
- Documentation: https://arxiv.org/abs/2007.03953
- Wiki page: https://iohprofiler.github.io/IOHanalyzer/
- Bug reports: https://github.com/IOHprofiler/IOHAnalyzer/issues
- Online service: http://iohprofiler.liacs.nl
- General Contact: mailto:iohprofiler@liacs.leidenuniv.nl
- Mailing List: https://lists.leidenuniv.nl/mailman/listinfo/iohprofiler
It provides:
- a web-based graphical user interface (GUI) to analyze and visualize the empirical performance of IOHs,
- interactive plotting,
- statistical evaluation,
- report generation, and
- a command-line interface (CLI) for the
R
console allowing for fine-grained controls.
It is built mainly on:
It is available through:
- a free online service that you can use right away, or
- the local installation of the package.
A free server http://iohprofiler.liacs.nl running the stable version of IOHanalyzer is hosted in Leiden Institute of Advanced Computer Science. You're welcome to check it out!
- [mandatory]
R
As IOHanalyzer is written as aR
package, theR
environment has to be installed first. The binary file and installation manual for R can be found here https://cran.r-project.org/. - [optional]
orca
required to download plotly figures. Please see https://github.com/plotly/orca for the installation instruction.
Please start up an R
console and install the stable version as:
R> install.packages('IOHanalyzer')
which is maintained on CRAN (Comprehensive R Archive Network).
The lastest development is always hosted on Github. In case you'd like to try out this version, the R
package devtool is needed:
R> install.packages('devtools')
R> devtools::install_github('IOHprofiler/IOHanalyzer')
If you want to run the version on which you develop:
R> devtools::install_git("/path/to/your/IOHanalyzer/git/repo")
The IOHanalyzer package can be loaded using the following commands:
R> library('IOHanalyzer')
R> runServer()
It should open a browser on the localhost
server, using a random port number. Of course, you
could also specify the port number directly:
R> runServer(port = 1234)
Have fun! For the complete reference on usage, please check out our Wiki page.
We provide a docker file for deploying IOHanalyzer on the server. Please see https://github.com/IOHprofiler/IOHanalyzer-docker for details.
Specific formats are required to load your benchmark data to IOHanalyzer. If your data sets are generated in the format of
- COCO/BBOB data format as regulated in https://hal.inria.fr/inria-00362649,
- Nevergrad data format (explained in https://github.com/facebookresearch/nevergrad), or
- IOHprofiler data format, which is motivated and modified from COCO data format,
then you just need to compress the data folder obtained from the experiment into a zip file and uploaded it. However, you are encouraged to convert your own benchmark data to the format regulated here!. The supported data format is specified in this page. Please follow the instruction there to convert your data sets.
- Hao Wang, Sorbonne Université, LIP6, France.
- Diederick Vermetten, Leiden Institute of Advanced Computer Science, The Netherlands.
- Furong Ye, Leiden Institute of Advanced Computer Science, The Netherlands.
- Ofer M. Shir, Tel-Hai College, Israel.
- Carola Doerr, Sorbonne Université, CNRS, LIP6, France.
- Thomas Bäck, Leiden Institute of Advanced Computer Science, The Netherlands.
When using IOHprofiler and parts thereof, please kindly cite this work as
Hao Wang, Diederick Vermettern, Furong Ye, Carola Doerr and Thomas Bäck: IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic, arXiv e-prints:2007.03953, 2020.
@ARTICLE{IOHprofiler,
author = {Hao Wang and Diederick Vermettern and Furong Ye and Carola Doerr and Thomas B{\"a}ck},
title = {{IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic}},
journal = {arXiv e-prints:2007.03953},
archivePrefix = "arXiv",
eprint = {2007.03953},
year = 2020,
month = July,
keywords = {Computer Science - Neural and Evolutionary Computing},
url = {https://arxiv.org/abs/2007.03953}
}
This application is governed by the BSD 3-Clause license.
BSD 3-Clause License
Copyright (c) 2018, All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
-
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
-
Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.