Automatic differentiation in C++; infinite differentiability of conditionals, loops, recursion and all things C++
###Abstract We provide an illustrative implementation of an analytic, infinitely-differentiable machine, implementing infinitely-differentiable programming spaces and operators acting upon them, as constructed in the paper Operational calculus on programming spaces. Implementation closely follows theorems and derivations of the paper, intended as an educational guide for those transitioning from automatic differentiation to this general theory.
The paper Implementing Operational calculus on programming spaces for Differentiable computing accompaning the implementation is avaliable on arXiv .
The dCpp project is constantly updated and optimized. This is the openSource version.
###Tutorial
dCpp is a flexible tool, allowing implementation and analysis of programs through operational calculus.
- Implementations of differentiable (sub) programs operating on differentiable derivatives of other (sub) programs, where the entire process may again be differentiable. This allows trainable training processes, and flexible program analysis through operational calculus.
- Operational calculus on programming spaces is the paper in which the theory is derived and the process of its use to the purpose of program analysis and deep learning is outlined.
- Implementing Operational calculus on programming spaces for Differentiable computing is the paper in which the implementation of this theory into dCpp is explained, where the reader is guided through the code and the theory simultaneously, as to better understand this flexible tool.
As the following tutorial is quite brief, please consult the discussions regarding common mistakes and solutions.
As most programmers face the need of differentiability through machine learning, we use the concept of logistic regression with softmax normalization as a vessel for this tutorial. We demostrate, how it is simply constructed using algorithmic control flow and reccursion, by employing dCpp.
First we include the necessities
#include <iostream>
#include <dCpp.h>
#include <vector>We initialize a n-differentiable programming space (order is arbitrary here)
using namespace dCpp;
int n_differentiable=2;
initSpace(n_differentiable);We will need the folowing functions
####Maps contained in the function algebra over var
By coding sigmoid(x), we will learn about creating differentiable maps, constructable using the differentiable programming space dCpp and the algebra of the virtual memory var.
var sigmoidMap(const var&v){return 1/(1+exp(-1*v));};
We test it out and and compute it on a simple example.
// set inputs
double x=4;
double y=2;
// set weights
var w_1(0.4);
var w_2(0.6);
// initialize weights as twice differentiable variables
dCpp::init(w_1);
dCpp::init(w_2);
// now we use sigmoid map as a differentiable map
var f=sigmoidMap(w_1*x+w_2*y);Accessing the derivatives is done by calling d(var* w) function of the class var. It returns the derivative with respect to the variable w, as a var variable.
// df/dw_1
std::cout<<"df/dw_1 = "<<f.d(&w_1).id<<std::endl;
// df/dw_2
std::cout<<"df/dw_2 = "<<f.d(&w_2).id<<std::endl;
// df/dw_1dw_1
std::cout<<"df/dw_1dw_1 = "<<f.d(&w_1).d(&w_1).id<<std::endl;
// df/dw_1dw_2
std::cout<<"df/dw_1dw_2 = "<<f.d(&w_1).d(&w_2).id<<std::endl;
// df/dw_2dw_1
std::cout<<"df/dw_2dw_1 = "<<f.d(&w_2).d(&w_1).id<<std::endl;
// df/dw_2dw_2
std::cout<<"df/dw_2dw_2 = "<<f.d(&w_2).d(&w_2).id<<std::endl;####Operator dTau
Similarly, we could have used the operator tau by coding , which allows one to create it's own elements of the differentiable programming space dCpp, returning a differentiable variable var. The operator tau is initialized by a double-->double map representing the desired function, and a var-->var map representing its derivative. Lets take a closer look, by creating a differentiable map log:var-->var.
//coding the derivative
var log_primitive(const var&v){return 1/v;};The map is created by providing the operator with the two maps
//operator returning a differentiable map
tau log(std::log,log_primitive);The map is now ready to use
// set variables
var x(10);
var y(20);
// initialize x and y as a differentiable variables
dCpp::init(x);
dCpp::init(y);
// now we use log as a differentiable map
var f=log(((x^2)-(y^0.23))^2.1);Again, we display all first and second derivatives
// df/dx
std::cout<<"df/dx = "<<f.d(&x).id<<std::endl;
// df/dy
std::cout<<"df/dy = "<<f.d(&y).id<<std::endl;
// df/dxdx
std::cout<<"df/dxdx = "<<f.d(&x).d(&x).id<<std::endl;
// df/dxdy
std::cout<<"df/dxdy = "<<f.d(&x).d(&y).id<<std::endl;
// df/dydx
std::cout<<"df/dydx = "<<f.d(&y).d(&x).id<<std::endl;
// df/dydy
std::cout<<"df/dydy = "<<f.d(&y).d(&y).id<<std::endl;####Integrating control structures
With dTau explained, we turn to coding the softmax normalization, we reveal how analytic virtual machines fully integrate control structures.
//simply code the map existing in the programming space dCpp
//and the belonging algebra
std::vector<var> softmax(const std::vector<var>& V){
std::vector<var> out;
var sum = 0.0;
for(const var &v:V){
sum+=exp(v);
}
for(const var &v:V){
out.push_back(exp(v)/sum);
}
return out;
}
We test it, by inititalizing a four-differentiable programming space and displaying all derivatives.
// initiaize Virtual memory of fourth order
initSpace(4);
//get a vector of variables
int size=2;
std::vector<var> vars;
for(int i=1;i<=size;i++){
var tmp=var(i);
init(tmp);
vars.push_back(tmp);
}
// use the softmax function
std::vector<var> f=softmax(vars);
// display derivatives of all four orders
// of one of the components
f[1].print();
####Integrating external libraries
Usage with external libraries written in generic paradigm is demonstrated on the example of Eigen. We will code a perceptron with sigmoid activations, followed by softmax normalization. We will use dCpp provided mappings in the dEigen header.
#include <iostream>
#include <dCpp.h>
#include <dEigen.h>
using namespace std;
using namespace dCpp;
//create a softmax function
template <typename Derived>
void softmax(Eigen::MatrixBase<Derived>& matrix){
//maps each element of the matrix by y=e^x;
dCpp::map_by_element(matrix,&dCpp::exp);
//sums the elements of the matrix using Eigens function
var tmp=matrix.sum();
//divides each element by the sum
for (size_t i=0, nRows=matrix.rows(), nCols=matrix.cols(); i<nCols; ++i)
for (size_t j=0; j<nRows; ++j)matrix(j,i)=matrix(j,i)/tmp;
}
void dEigenExample(){
// space is n-times differentiable
int n=2;
// initialize the space
dCpp::initSpace(n);
// Matrix holding the inputs (imgSizeX1 vector)
const int inputSize=28;
Eigen::Matrix<var,1,inputSize>input=Eigen::Matrix<var,1,inputSize>::Random(1,inputSize);
dCpp::init(input);
// number of outputs of the layer
const int outputSize=1;
// matrix of weights on the first level (imgSizeXnumOfOutOnFirstLevel)
Eigen::Matrix<var,inputSize,outputSize>firstLayerVars=Eigen::Matrix<var,inputSize,outputSize>::Random(inputSize,outputSize);
// initializing weights
dCpp::init(firstLayerVars);
// mapping of the first layer
Eigen::Matrix<var,outputSize,1>firstLayerOutput=input*firstLayerVars;
// add bias
Eigen::Matrix<var,outputSize,1>bias=Eigen::Matrix<var,outputSize,1>::Random(outputSize,1);
// initialize bias
dCpp::init(bias);
firstLayerOutput=bias+firstLayerOutput;
// apply sigmoid we coded earlier
dCpp::map_by_element(firstLayerOutput,sigmoid);
// apply sofmax layer
softmax(firstLayerOutput);
// display the first output layer and its (1-n)-th derivatives
for (size_t i=0, nRows=firstLayerOutput.rows(), nCols=firstLayerOutput.cols(); i<nCols; ++i){
for (size_t j=0; j<nRows; ++j) firstLayerOutput(j,i).print();
cout<<endl;
}
}###Citation If you use dCpp in your work, please cite the following paper
Žiga Sajovic: Implementing Operational calculus on programming spaces for Differentiable computing. arXiv e-prints arXiv:1612.0273 (2016)
A BibTex snippet has been provided for your convenience
@article{
Author = {Žiga Sajovic},
Title = {Implementing Operational calculus on programming spaces for Differentiable computing},
journal = {arXiv e-prints},
Year = 2016,
volume = {arXiv:1612.0273},
Eprint = {1612.02731},
Eprinttype = {arXiv},
}
dC++ by Žiga Sajovic is licensed under a Creative Commons Attribution 4.0 International License.
