"Manet" is a library for MAC networks.
MAC networks are networks consisted only by multiply–accumulate operations, and they are activation-free and can be used as building blocks for deep neural networks.
Multiply–accumulate operation is a generalization of the perceptron and the linear neuron. But surprisingly, it can be the only mechanism to construct deep neural networks. The non-linearity can be eliminated, we can easily see the point that:
- exponential functions is pure multiplicative
- a wide range of non-linear functions can be constructed by using different combinations of multiplications and additions.
It also owns a beautiful mathematical background from the field of arithmetic expression geometry.
We demonstrate the performance of multiply–accumulate network on MNIST dataset.
We gradually replace the non-linear activation functions with MAC operations, and the performance is almost the same and sometimes is even better. Please check mnist0.py to mnist3.py for details.
python -m demo.mnist.train -m mnist0
python -m demo.mnist.train -m mnist1
python -m demo.mnist.train -m mnist2
python -m demo.mnist.train -m mnist3Note: mnist3.py is buggy and conv replacement is still ongoing.