FinDi: Finite Difference Gradient Descent can optimize any function, including the ones without analytic form, by employing finite difference numerical differentiation within a gradient descent algorithm.
- Free software: MIT license
- Documentation: https://findi-descent.readthedocs.io/en/latest/
A preferred method to install findi
is through Python's package installer pip. To install findi
, run this command in your terminal
pip install findi-descent
Alternatively, you can install the package directly from GitHub:
git clone -b development https://github.com/draktr/findi-descent.git
cd findi-descent
python setup.py install
Finite Difference Gradient Descent (FDGD) is a modification of the regular GD algorithm that approximates the gradient of the objective function with finite difference derivatives, as
Analogously, the FDGD update rule is given as
where
descent()
- regular FDGD algorithmpartial_descent()
- FDGD algorithm where in each epochparameters_used
number of parameters are randomly selected to be differenced. This approach is useful in cases where the evaluation of objective function is computationally expensivepartially_partial_descent()
- FDGD algorithm that usespartial_descent()
algorithm for the firstpartial_epochs
number of epochs anddescent()
for the remaining epochs. This approach is useful as it combines the computational efficiency ofpartial_descent()
with the approximational accuracy ofdescent()
- Numba mode - Numba just-in-time compilation is available for all algorithms, including automatically parallelized evaluation. This drastically decreases computation time, however, it also requires the objective function to be Numba-compiled
joblib
parallelization - supported in Python mode. This is helpful, especially with high-dimensional problems where Numba objective function is unfeasiblevalues_out()
function - exports outputs, parameters, and constants values for each epoch asPandas
DataFrame
. This is useful for, among other things, algorithm convergence analytics and hyperparameter (e.g. learning rates and differences) tuning- Variable learning rates and difference values - other than scalars,
l
andh
arguments also accept array_like structures (e.g. lists andNumpy
arrays). These can be constructed manually, by the library OptSchedule which provides a variety of decay schedules momentum
hyperparameter - accelerates gradient descent in the relevant direction and dampens oscillations.momentum = 0
(default value) implies no acceleration and dampening. The update rule withmomentum > 0
is
- support for metaparameters - FinDi accepts objective functions that require metaparameters to be passed to it. By metaparameter is considered any parameter passed to the objective function that will not be differenced and its value will be held constant throughout epochs
- support for multiple outputs - FinDi accepts objective functions that return more than one value. For example, if the objective function has a convex optimization routine within it, FinDi allows for the objective function to return the regular objective value along with the solutions to the optimization problem. The first value of the return structure will be taken as the objective value to be minimized
- Optimizing objective functions that cannot be expressed or solved analytically or discontinuous functions
- Intuitive and easy to communicate its implementation, unlike most of the derivative-free optimization methods
- Convenient work with blackbox or proprietary objective functions through metaparameters, where source code might be inaccessible
- Increased computational efficiency with Numba just-in-time compilation
- Supports parallelization via
joblib
ornumba
library - Partial Gradient Descent makes high-dimensional, simple problems less computationally expensive to solve
- Built-in support for variable learning rates and differences
Below is a simple demonstrative example to show how to use findi
. More examples can be found in the documentation, including the examples of problems that can be solved by findi
and not by other Python Gradient Descent implementations.
import findi as fd
# Defining the objective function
def foo(params):
return [(params[0]+2)**2]
# Descent
outputs, parameters = fd.descent(
objective=foo,
initial=5,
h=0.0001,
l=0.01,
epochs=1000,
)
print("Solution (argmin): ", parameters[-1])
print("Objective value at solution (min): ", outputs[-1])
# Saves values of outputs and parameters as Pandas DataFrame...
values = fd.values_out(outputs, parameters, columns=["x"])
# ...to be stored as a CSV file
values.to_csv("values.csv")
- Easy to be understood and used by those with little computer science background, including scientists, researchers and industry practitioners
- Flexibility for proprietary modifications
- Emphasis on computational efficiency
- Use consistency across approaches (Numba vs Python, regular Gradient Descent vs Partial Gradient Descent etc.)
- Tested
- Dedicated and detailed technical and applicative documentation
- Formatting deferred to Black
Feature requests are more than welcome through the Issues forum, as are bug reports and improvement recommendations. Anyone is more than welcome to become a contributor or maintainer.