lnshi / ml-exercises

Have some fun with ML. 🕵🤖🧠🤔

Project GitHub Pages

• Leonard's Machine Learning Exercises

Topics

ml_basics

• Multivariable linear regression(gradient descent)

• Gradient and gradient descent

  • Derivative
  • Derivative and partial derivative
  • Derivative and directional derivative
  • Derivative and gradient
  • Gradient descent algorithm

• Gradient descent learning rate chosen

• Normal equation

  • Vector addition and subtraction
  • Vector dot product (scalar product, inner product)
  • Vector cross product
  • Normal equation

• PDF vs PMF vs CDF

• Bayes’ Theorem and MLE MAP

• Logistic regression (binomial regression) and regularization

  • Experience scipy.optimize.fmin_tnc
  • Regularization
  • Norm of vector and matrix
  • Dataset features expansion/extraction
  • When a lower dimensional space NOT discriminable dataset is PROJECTED to a PROPER higher dimensional space it always will be discriminable, the boundary is a hyper plane or just a discrimination function.
  • Model accuracy comparison between 10-dimensional and 6-dimensional
  • 'linear_model.LogisticRegression' with sklearn

• GLM and exponential family distributions

  • Bernoulli distribution in GLM form
  • Gaussian distribution (normal distribution) in GLM form
  • Softmax regression (multinomial logistic regression) (categorical distribution (variant 3)) in GLM form
  • GLM ⇒ linear regression
  • GLM ⇒ logistic regression
  • Why the PMF for categorical distribution(special form of multinomial distribution: k > 2 and n = 1) has no coefficient like the multinomial distribution's PMF
  • How to use the table here to build GLM quickly

• Multinomial logistic regression

• Neural network (hand-written digits recognition)

  • What's a neural network
  • One-hot encoding
  • Forward propagation
  • Backpropagation algorithm

• SVM

  • Functional margin vs geometric margin, and SVM interpretation
  • Lagrange multiplier, Lagrange duality and KKT conditions
  • Coordinate ascent algorithm and SMO
  • Slack variables and penalty factors

• PCA

  • What does PCA do?
  • Another interpretation of what does PCA do
  • What are the application scenarios of PCA?
  • PCA pseudo code
  • Eigenvalues and eigenvectors
  • Singular value and Singular Value Decomposition
  • PCA dimensionality reduction example

• Some generic questions

  • 1. Type 1 error, type 2 error, power of test
  • 2. Over-fitting, under-fitting
  • 6. How is k-NN different from k-means clustering?
  • 7. p-value
  • 12. What is naive in a Naive Bayes?
  • 13. How can you select k for k-means?
  • 14. How to measure a model?
  • 18. What is cross-validation?
  • 19. What is bias-variance trade off?
  • 20. The types of biases that occur during sampling?
  • 21. What is the confusion matrix?
  • 22. What are exploding/vanishing gradients?
  • 25. What is eigenvectors and eigenvalues?
  • 27. What are AutoEncorders?
  • 28. How do you avoid the over-fitting during the training?
  • 29. What are the differences among standardization, normalization and regularization?
  • 30. Some feature selection methods used to select the right variables
  • 33. MAE vs MSE vs RMSE
  • 34. How can outlier values be treated?
  • 36. What is a Box-Cox Transformation?
  • 37. What is the hyperbolic tree?
  • 39. What's the difference between probability and likelihood?
  • 40. What cross-validation technique would you use on a time series dataset?

• Decision tree and random forest

• k-fold cross validation sklearn example

• Ensemble learning

  • What is bootstrapping (bootstrap sampling)?
  • Why do we need bootstrap sampling?
  • Implement Bootstrap Sampling in Python
  • What is ensemble learning
  • Boosting
  • Bagging
  • Stacking

• Gradient(GBM) vs AdaBoost vs XGBoost

  • Decision tree vs random forest
  • Random forest vs gradient boosting
  • Gradient vs AdaBoost
  • XGBoost

• RNN and LSTM

• Model parameters vs hyperparameters

• Optimizers

  • Gradient descent
  • Types of gradient descent
  • Role of an optimizer
  • Types of optimizers
  • Momentum
  • Nesterov accelerated gradient(NAG)
  • Adagrad — Adaptive Gradient Algorithm
  • Adadelta
  • RMSProp
  • Adam — Adaptive Moment Estimation
  • Nadam- Nesterov-accelerated Adaptive Moment Estimation

Questions

1. In gradient descent, must there be a learning rate transition point(safety threshold) for all kinds of cost functions?

2. How do we extend this to the cross product of a four dimensional vector or more higher, like the right part of the above graph?

3. When a lower dimensional space NOT discriminable dataset is PROJECTED to a PROPER higher dimensional space it always will be discriminable, the boundary is a hyper plane or just a discrimination function, what are the differences of the 'a hyper plane' or 'a discrimination function' here?

4. What are the best practices / skills / underlying theories for the features expansion/extraction?

5. For the hand-written digits recognition NN why the hidden layer input size is 25? How about design to use a different size? What is the theory/best practices/skills/prior knowledges here?

Accumulations / References

1. 如何理解最小二乘法？

2. np.array([0, 0]) vs np.array([0., 0.])

   >>> import numpy as np
   >>> t = np.array([0, 0])
   >>> t[0] = 0.97
   >>> t
   array([0, 0])
   >>> t[0] = 1.97
   >>> t
   array([1, 0])
   >>> t = np.array([0., 0.])
   >>> t[0] = 0.97
   >>> t
   array([0.97, 0.  ])
   >>> t = np.array([0, 0.])
   >>> t[0] = 0.97
   >>> t
   array([0.97, 0.  ])
   

Memos

1. Exponential family form of multinomial distribution <- my answer -> link to topic 'GLM and exponential family distributions'

2. Batch Normalization Tensorflow Keras Example