songweige / CNTK

Convolutional Neural Tangent Kernel

Convolutional Neural Tangent Kernel (CNTK)

This repository contains the code for Convolutional Neural Tangent Kernel (CNTK) in the following paper. I have made some modifications to run it on different datasets.

On Exact Computation with an Infinitely Wide Neural Net (NeurIPS 2019)

Citation

@inproceedings{arora2019exact,
  title={On exact computation with an infinitely wide neural net},
  author={Arora, Sanjeev and Du, Simon S. and Hu, Wei and Li, Zhiyuan and Salakhutdinov, Ruslan and Wang, Ruosong},
  booktitle={Thirty-third Conference on Neural Information Processing Systems},
  year={2019}
}

Usage

Require Python 2.7 and CUDA.

1. Install CuPy.
2. Download CIFAR-10.

wget https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz; tar zxvf cifar-10-python.tar.gz

3. Parallelize Line 143-146 in CNTK.py according to your specific computing enviroment to utilize multiple GPUs.

To reproduce results in Table 1 in our paper:

For column CNTK-V:

python CNTK.py --gap no --fix no --depth DEPTH

where DEPTH is 3, 4, 6, 11 or 21.

For column CNTK-GAP:

python CNTK.py --gap yes --fix yes --depth DEPTH

where DEPTH is 3, 4, 6, 11 or 21.