Regression

Regression is used to find a model for continuous variables. The model can be used to handle prediction, controling and causal link task. Before doing regression, we need to determine which regression function is good for the sample data e.g. linear, polynomial or parabolic.

Develop tools and techniques

Python
Pycharm

Regression closed form

Variable transformation: use linear regression to solve non-linear model
Find regression parameter

Practice

Gradient descent to solve linear regression

In ML Lecture 1, I follow the tutorial and practice to use gradient descent to solve linear regression.

Without using the linear regression closed form
If using the pure gradient descent, it will take too much time to find the optimal parameter. Therefore, we need to give different learning rates to b and w, and update them in each iteration. The cool method is called Adagrad.
Adagrad: I remeber my professor tell us that it can normalize the gradient at each iteration by gradient sd


Pure gradient descent	Gradient descent with Adagrad

Regression closed form for multiple dimensional vector

We have m data tuples which contain vector x and scalar y,

x is a n-dimensional vector,

Now, Assume y has linear relationship with x,

For simplifying the notation, we combine α and all β into a vector β, all y into vector y. We create a maxtrix X whose row vector ia vector x and added 1 at first column.

We assume the linear regression model is

To minimize error, the loss function is

Xβ can be viewd as a hyperspace in R^m spanned by the n+1 columns of X. X is fixed and β is variable. We want to find the best β so that the error will be minimized. Now, we imagine that if all errors are 0, ŷ is the projection of y onto the hyperplane spanned by X

Obviously, y-ŷ is orthogonal to the columns of X, we can utilize this property to get the best β instead of solving directly the loss function by partial differential,