jncraton / matrix-optimization

Matrix Optimization

This project provides a matrix addition and matrix multiplication subroutine. These subroutines are currently suboptimal.

Matrix Addition

Matrix addition results in a new matrix that is the sum of corresponding entries from two matrices. Here is an example:

The basic algorithm for implementing matrix addition is:

for row in matrix:
  for column in matrix:
    c[row][col] = a[row][col] + b[row][col]

This algorithm requires iterating over the columns while iterating over the rows, so it is O(n^2) with respect to the size of the matrix.

The implementation for this algorithm is found in the add function in matrix.c.

Matrix Multiplication

Matrix multiplication can be defined as:

Each element in the resulting matrix will be the sum of the products in a row from matrix a and a column from matrix b.

An algorithm for matrix multiplication would be:

for row in matrix:
  for column in matrix:
    c[row][col] = 0
    for i in range(colsize):
      c[row][col] += a[i][col] + b[row][i]

This algorithm requires iterating over the columns while iterating over the rows and then iterating over the items within an entire row and column, so it is O(n^3) with respect to the size of the matrix. Note that this is not an optimal algorithm. The optimal algorithm is an area of ongoing research, with the current state-of-the-art being O(n^2.37286).

The implementation for this algorithm is found in the mul function in matrix.c.

Memory and Cache Optimization

These algorithms do not make optimal use of the way that CPUs and memory interact in modern systems. Both algorithms can be adjusted to have better cache hit rates and memory properties. Addition can trivially be improved by loop interchange while multiplication can be improved using loop tiling.