Oblique Decision Trees from Derivatives of ReLU Networks (Locally Constant Networks)

This repository is for the paper

"Oblique Decision Trees from Derivatives of ReLU Networks" by Guang-He Lee and Tommi S. Jaakkola in ICLR 2020.
The old title for this paper is "Locally Constant Networks"

Package version

linux (we only tested the codes on Ubuntu)
python3.6
pytorch1.1.0
scikit-learn (for computing AUC)

Repo structure

scripts/: scripts for reproducing the experiments
data/: datasets used in the paper. Training/validation/testing splits are provided.
log/: the logs of the training results will be stored here.
checkpoint/: the checkpoint of the learned model will be stored here.

Main files

run_net_training.py: main python file for running LCNs/ALCNs/LLNs
run_elcn_training.py: main python file for running ELCN

Quick start and reproducing experiments

Example scripts are available in scripts/. For example,

scripts/bace_split/, scripts/HIV_split/, and scripts/PDBbind/ contain the scripts for reproducing the Bace, HIV, and PDBbind experiments with the tuned parameters. For example, you may run
```
 sh scripts/bace_split/run_LCN.sh
```

which will run the LCN model with the Bace dataset.

scripts/tox21_split/ and scripts/sider_split/ contain the scripts for reproducing the Tox21 and Sider experiments with the parameter tuning procedure. Since the two datasets are multi-label, you have to specify the task number when you run the script. For example, you may run
```
 sh scripts/sider_split/run_LCN.sh 1
```

which will run the LCN model with the the 1st label in the Sider dataset. The Tox21 contains 12 labels, and the Sider dataset contains 27 labels.

The results will be stored in the log/ directory. The last 3 columns record the training, validation, and testing performance, respectively (from left to right).

How to set hyper-parameters

Switching among LCN, ALCN, and LLN

The default model in run_net_training.py is LCN.

Switching from LCN to ALCN: setting --anneal to approx.
Switching from LCN to LLN: setting --net_type to locally_linear.

Some suggestions for hyper-parameter tuning

If you would like to apply the codes for other datasets, we suggest to tune the following hyper-parameters. You can see the complete list of hyper-parameters in arg_utils.py.

Depth of the network (--depth).
We suggest to use DropConnect (set --drop_type to node_dropconnect) and mildly tune the dropping probability (e.g., try --p in {0.25, 0.5, 0.75}).
You can start trying the model by setting --back_n (the depth of the network g_φ) to 0. If it doesn't work, please try to increase it. In our experiments, we found that we need to increase it for regression tasks, and we can simply keep it to 0 for classification tasks.
You may want to tune the learning iterations (--epochs), learning rate (--lr), and optimizer (--optimizer) for your tasks. If you change the learning iterations (--epochs), you probably should also change the --lr_step_size and --gamma (see their meaning in the help descriptions in arg_utils.py).
You may enlarge the --batch-size to accelerate training.

ELCN

If you would like to try the ensemble version (ELCN), you can just specify the maximum ensemble number as the --ensemble_n, and the codes will automatically stores all the logs for each ensemble iteration. For example, if you would like to tune the ensemble number in {1,2,4,8}, you can just run --ensemble_n 8 once, and check the logs for the results with ensemble numbers {1,2,4,8}.

You may also tune the shrinkage parameter. We suggest to try --shrinkage in {1, 0.1}.

Contact

Guang-He Lee (guanghe@csail.mit.edu)

Citation:

If you find this repo useful for your research, please cite the paper

@inproceedings{lee2020oblique,
	title={Oblique Decision Trees from Derivatives of ReLU Networks},
	author={Guang-He Lee and Tommi S. Jaakkola},
	booktitle={International Conference on Learning Representations},
	year={2020},
	url={https://openreview.net/forum?id=Bke8UR4FPB}
}

guanghelee / iclr20-lcn