stonlimart / tiny-cnn

Fork of Tiny CNN, a C++11 convolutional neural network library.

Tiny CNN: A C++11 Convolutional Neural Network Library

This is a fork of nyanp/tiny-cnn. Minor modifications are planned and CMake is used for building.

Index

• Design Principles
• Supported Networks
  • Layer Types
  • Activation Functions
  • Loss Functions
  • Optimization Algorithms
• Dependencies
• Building
• Examples
• Useful References
• License

Design Principle

• Fast - without GPU
  • TBB threading and SSE/AVX vectorization
  • 98.8% accuracy on MNIST [2] in 13 minutes training (@Core i7-3520M)
• Header only, policy-based design
• Small dependency & simple implementation

Supported Networks

Layer Types

• Fully-connected layer

• Fully-connected layer with dropout [1]

• Convolutional layer

• Average pooling layer

  [1] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. Computing Research Repository, abs/1207.0580, 2012.

Activation Functions

• Hyperbolic tangent
• Sigmoid
• Rectified linear
• Identity

Loss Functions

• Cross-entropy
• Mean-squared-error

Optimization Algorithms

• Stochastic gradient descent (with/without L2 normalization)
• Stochastic gradient Levenberg-Marquardt

Dependencies

• Boost
• Intel TBB

Building

This fork of Tiny CNN is built using CMake (tested on Ubuntu 14.04):

$ sudo apt-get install build/essential
$ sudo apt-get install cmake
$ sudo apt-get install libboost-all-dev
$ sudo apt-get install libtbb-dev
$ git clone https://github.com/davidstutz/tiny-cnn
$ cd tiny-cnn
$ mkdir build
$ cd build
$ cmake ..
$ make

Adapt config.h for using TBB, SSE or AV.

Note: Depending on your installation of Boost and TBB, you may need to adapt cmake/FindTBB.cmake and CMakeLists.txt!

Examples

Examples can be found in src/main.cpp. The MNIST [2] dataset is available online at http://yann.lecun.com/exdb/mnist/.

[2] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner.
    Gradient-based learning applied to document recognition.
    Proceedings of the IEEE, 86(11):2278-2324, November 1998.

Useful References

Some useful references with working with (deep) convolutional neural networks. Further references can also be found on davidstutz.de/neural-networks-reading-list and [deeplearning.net] (http://deeplearning.net/).

[3] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel.
    Backpropagation applied to handwritten zip code recognition.
    Neural Computation, 1(4):541–551, 1989.
[4] C. Bishop.
    Pattern Recognition and Machine Learning.
    Springer Verlag, New York, 2006.
[5] Y. Bengio.
    Learning deep architectures for AI.
    Foundations and Trends in Machine Learning, (1):1–127, 2009.
[6] K. Jarrett, K. Kavukcuogl, M. Ranzato, Y. LeCun.
    What is the best multi-stage architecture for object recognition?
    In Computer Vision, International Conference on, pages 2146–2153, 2009.
[7] Y. LeCun, K. Kavukvuoglu, C. Farabet.
    Convolutional networks and applications in vision.
    In Circuits and Systems, International Symposium on, pages 253–256, 2010.
[8] A. Krizhevsky, I. Sutskever, and G. E. Hinton.
    ImageNet classification with deep convolutional neural networks.
    In Advances in Neural Information Processing Systems, pages 1097–1105, 2012

license

The BSD 3-Clause License, see nyanp/tiny-cnn.