ujjwalkhandelwal / Dual-numbers-and-automatic-differentiation-using-Python

Implemented the forward mode of automatic differentiation with the help of dual numbers using Python.

Dual numbers and automatic differentiation using Python

Implemented the forward mode of automatic differentiation with the help of dual numbers. We first implemented a class Dual with the constructor init, the functions add, radd, sub, rsub, mul, rmul, truediv, rtruediv, neg and pow. As the names suggest, those functions and properties implement basic arithmetic operations for Dual numbers:

init : constructor that initialises an object of class Dual. Each object represents a dual number a+εb with real component a (self.real) and dual component b (self.dual).

add : adds an argument argument to the dual number, i.e. a + εb + argument.

radd : adds the dual number to the argument argument, i.e. argument + a + εb.

sub : subtracts an argument argument from the dual number.

rsub : subtracts the dual number from the argument argument.

mul : multiplies the dual number with the argument argument.

rmul : multiplies an argument argument with the dual number.

truediv : divides the dual number by an argument argument.

rtruediv : divides the argument argument by the dual number.

neg : returns the negative of the dual number a + εb, i.e. -a - εb.

pow : takes the power-th power of the dual number. i.e. (a + εb)^power

Next, we implemented the following functions that are acting on dual numbers of the form a+εb:

log_d : log(a+εb)

exp_d : exp(a+εb)

sin_d : sin(a+εb)

cos_d : cos(a+εb)

sigmoid_d : 1/1+exp(−(a+εb))

Dependencies

• Numpy (pip install numpy)

Utilities

Once the installation is finished (downloading or cloning the files), go to the dual folder and follow the below simple guidelines to execute Dual class effectively (either write the code in command line or in a python editor with the name say main.py) OR you can also follow the jupyter notebook with the name dual.ipynb.

>>> import numpy as np
>>> from ad_dual import Dual

Next, import the functions (not necessarily all the functions but the one you need) using:

>>> from func import log_d, exp_d, sin_d, cos_d, sigmoid_d

Example-1

At x=1 and y=2,

f = 1, f_x = -13, f_y = 4

x = Dual(real=1, dual={'x': 1})
y = Dual(real=2, dual={'y': 1})

f = (x**3) - 2*(x**2)*(y**2) + (y**3)
print(f)

You will see the following output:

f = 1
fx = -13
fy = 4

NOTE: The key, value pair in the dictionary indicates the symbol with which you want to represent the variable and the value of the dual number respectively. Like y = Dual(real=2, dual={'y': 7}) represents y = 2 + 7ε. In case you want to calculate the partial derivatives of f, keep the value of the dict as 1 (y = Dual(real=2, dual={'y': 1})

Example-2

At x=2 and y=4,

f = 9, f_x = 4, f_y = -4

x = Dual(real=2, dual={'x': 1})
y = Dual(real=4, dual={'y': 1})
f = 81*x / (x+(y**2))
print(f)

You will see the following output:

f = 9.0
fx = 4.0
fy = -4.0

Example-3

At x=1, y=2 and z=1,

f = 6, f_x = 5, f_y = -4, f_z = 4

x = Dual(1, {'x': 1})
y = Dual(2, {'y': 1})
z = Dual(1, {'z': 1})

f = 36*x*z / (x+(z**2)+(y**2))
print(f)

You will see the following output:

f = 6.0
fx = 5.0
fz = 4.0
fy = -4.0

Example-4

At x=π and y=π,

f = 0, f_x = 1/(1-π²) = −0.112744, f_y = 0

x = Dual(np.pi, {'x': 1})
y = Dual(np.pi, {'y': 1})

f = sin_d(x)/(cos_d(y)+(x**2))
print(f)

You will see the following output:

f = 0.0
fx = -0.112745
fy = 0.0

Back Propagation using Dual Numbers

Work is in progress and it will soon be updated...