README file for PROBEN1 benchmark collection: ================================================== Contents of the PROBEN benchmark set directory: ------------------------------------------------- This directory contains datasets to be used for neural network training. All data files use the same very simple format. Each dataset is usually in its own subdirectory along its documentation. The following directories exist: building/ Building enegery comsumption prediction problem. (Data of the "Great energy predictor shootout" contest) cancer/ Wisconsin breast cancer diagnosis problem from UCI machine learning database. card/ Credit card approval problem from UCI machine learning database. diabetes/ Diabetes diagnosis problem from UCI machine learning database doc/ Contains the techreport containing documentation of datasets and rules and conventions for their use. flare/ solar flare prediction problem from UCI machine learning database gene/ gene splice-junction detection problem from UCI machine learning database glass/ Glass type identification problem from UCI machine learning database. heart/ heart-disease diagnosis problem from UCI machine learning database. mushroom/ mushroom edible/poisonous classification problem from UCI machine learning database. This directory is present only in the ADDENDUM to PROBEN1, since the mushroom problem is very large and not otherwise very interesting. soybean/ Soybean disease classification problem from UCI machine learning database. thyroid/ Thyroid normal/super/sub-function diagnosis problem from UCI machine learning database. Scripts/ Directory with some utility scripts for those who want to prepare their own datasets. Not needed when one only wants to use PROBEN1 without changing it. ======================================================================== Quick overview of the size of the datasets: which #attrib #in #out #examples ------------------------------------ building 6 14 3a 4208 cancer 9 9 2c 699 card 15 51 2c 690 diabetes 8 8 2c 768 flare 9 24 3a 1066 gene 60 120 3c 3175 glass 9 9 6c 214 heart 13 35 2c 920 heartc 13 35 2c 303 hearta 13 35 1a 920 heartac 13 35 1a 303 horse 20 58 3c 364 mushroom 22 125 2c 8124 soybean 35 82 19c 683 thyroid 21 21 3c 7200 c = class outputs(0/1), a = analog outputs(0...1) The heart, hearta, heartc, and heartac datasets are all in the heart directory (see there for documentation). ======================================================================== Description of data file format: The following is what a data file looks like (example from glass1.dt): bool_in=0 real_in=9 bool_out=6 real_out=0 training_examples=107 validation_examples=54 test_examples=53 0.281387 0.36391 0.804009 0.23676 0.643527 0.0917874 0.261152 0 0 1 0 0 0 0 0 0.260755 0.341353 0.772829 0.46729 0.545966 0.10628 0.255576 0 0 0 1 0 0 0 0 [further data lines deleted] Each line after the header lines represents one example; first the examples of the training set, then validation set, then test set. The sizes of these sets are given in the last three header lines (the partitioning is always 50%/25%/25% of the total number of examples). The first four header lines describe the number of input coefficients and output coefficients per example. A boolean coefficient is always represented as either 0 (false) or 1 (true). A real coefficient is represented as a decimal number between 0 and 1. For all data sets, either bool_in or real_in is 0 and either bool_out or real_out is 0. Coefficients are separated by one or multiple spaces; examples (including the last) are terminated by a single newline character. That's all. The datafiles of problem xx are named xx1.dt, xx2.dt, and xx3.dt; they are located in directory proben1/xx. The only difference between the three versions is the ordering of the examples (so that different examples are in the training, validation, and test set). ======================================================================== I suggest that you start by having a look at the techreport in the doc/ directory. Most of the datasets are from the UCI machine learning databases archive (available by anonymous ftp to ics.uci.edu [128.195.1.1] in directory /pub/machine-learning-databases). This archive is maintained by Patrick M. Murphy and David W. Aha. Many thanks to them for their valuable service. The databases themselves were donated by various researchers -- see the documentation files in the individual dataset directories for details. Most data sets in the UCI repository are represented in symbolic form and are meant to be used with symbolic machine learning algorithms. What I have done is the following. - I selected data sets that seemed suitable to neural network learning, - For each of them, I decided on an attribute representation - For each of them, I wrote a script to convert into this attribute representation, using an exactly identical target format for all data sets. - I conducted some experiments with each of the data sets in order to find ball park figures of how good the results should be that one obtains when using the data sets. - I wrote a report describing the data sets, the results, and a set of rules to be applied when using the data sets and when publishing results. The goals of the whole project are - to give NN researchers easier access to a number of data sets representing real problems. - to make published results better reproducible. - to make published results directly comparable. - to decrease the frequency of methodological errors in NN benchmarking. Lutz Lutz Prechelt (prechelt@ira.uka.de) Department of Informatics University of Karlsruhe D-76128 Karlsruhe Germany ========================================================================