sangwoomo / ml-resources

Personal and biased selection of ML resources

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Some resources for ML research

Personal and biased selection of ML resources.

Disclaimer: I'm a noivce in ML research, and I read only a few of the list.

Table of Contents

Beginner's Guide

Must Read

  • Machine Learning: A Probabilistic Perspective (Murphy)
  • Deep Learning (Goodfellow et al.)
  • Reinforcement Learning: An Introduction (Sutton & Barto)

Recommended

  • Convex Optimization (Boyd & Vandenberghe)
  • Graphical Models, Exponential Families, and Variational Inference (Wainwright & Jordan)
  • Learning from Data (Abu-Mostafa) -> for whom interested in learning theory

Recent Topics

  • Read research blogs (e.g., OpenAI, BAIR, CMU)
  • Read lectures from Berkeley, Stanford, CMU or UofT (e.g., unsupervised learning)
  • There are lots of good sources, but I stop updating them up-to-date

Machine Learning

There are many ML books, but most of them are encyclopedic.
I recommend to take a course using Murphy or Bishop book.

Textbook

  • Machine Learning: A Probabilistic Perspective (Murphy) ✨
  • Pattern Recognition and Machine Learning (Bishop) ✨
  • The Elements of Statistical Learning (Hastie et al.)
  • Pattern Classification (Duda et al.)
  • Bayesian Reasoning and Machine Learning (Barber)

Lecture

Deep Learning

Goodfellow et al. is the new classic.
For vision and NLP, Stanford lectures would be helpful.

Textbook

  • Deep Learning (Goodfellow et al.) ✨

Lecture (Practice)

Lecture (Theory)

Generative Model

I seperated generative model as an independent topic,
since I think it is big and important area.

Lecture

Reinforcement Learning

For classic (non-deep) RL, Sutton & Barto is the classic.
For deep RL, lectures from Berkeley/CMU looks good.

Textbook

  • Reinforcement Learning: An Introduction (Sutton & Barto) ✨

Lecture

Graphical Model

Koller & Friedman is comprehensive, but too encyclopedic.
I recommend to take an introductory course using Koller & Friedman book.

Wainwright & Jordan only focuses on variational inference,
but it gives really good intuition for probabilistic models.

Textbook

  • Probabilistic Graphical Models: Principles and Techniques (Koller & Friedman)
  • Graphical Models, Exponential Families, and Variational Inference (Wainwright & Jordan) ✨

Lecture

Optimization

Boyd & Vandenberghe is the classic, but I think it's too boring.
Reading chapter 2-5 would be enough.

Bertsekas more concentrates on convex analysis.
Nocedal & Wright more concentrates on optimization.

Textbook

  • Convex Optimization (Boyd & Vandenberghe) ✨
  • Convex Optimization Theory (Bertsekas)
  • Numerical Optimization (Nocedal & Wright)

Lecture

Learning Theory

In my understanding, there are two major topics in learning theory:

  • Learning Theory: VC-dimension, PAC-learning
  • Online Learning: regret bound, multi-armed bandit

For learning theory, Kearns & Vazirani is the classic; but it's too old-fashined.
Abu-Mostafa is a good introductory book, and I think it's enough for most people.

For online learning, Cesa-Bianchi & Lugosi is the classic.
For multi-armed bandit, Bubeck & Cesa-Bianchi provides a good survey.

Textbook (Learning Theory)

  • Learning from Data (Abu-Mostafa) ✨
  • Foundations of Machine Learning (Mohri et al.)
  • An Introduction to Computational Learning Theory (Kearns & Vazirani)

Textbook (Online Learning)

  • Prediction, Learning, and Games (Cesa-Bianchi & Lugosi)
  • Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems (Bubeck & Cesa-Bianchi)

Lecture

Statistics

Statistics is a broad area; hence, I listed only a few of them.
For advanced topics, lectures from Berkeley/Stanford/CMU/MIT looks really cool.

Textbook (Statistical Inference)

  • All of Statistics (Wasserman)
  • Computer Age Statistical Inference (Efron & Hastie) ✨
  • Time Series Analysis and Its Applications: With R Examples (Shumway & Stoffer)

Textbook (Nonparametrics)

  • All of Nonparametric Statistics (Wasserman)
  • Introduction to Nonparametric Estimation (Tsybakov)
  • Gaussian Process and Machine Learning (Rasmussen & Williams) ✨
  • Bayesian Nonparametrics (Ghosh & Ramamoorthi) ✨

Textbook (Advanced Topics)

  • High-Dimensional Statistics: A Non-Asymptotic Viewpoint (Wainwright) ✨
  • Statistics for High-Dimensional Data (Bühlmann & van de Geer)
  • Asymptotic Statistics (van der Vaart)
  • Empirical Processes in M-Estimation (van der Vaart)

Lecture

Topics in Machine Learning

Miscellaneous topics related to machine learning.
There are much more subfields, but I'll not list them all.

Information Theory

  • Elements of Information Theory (Cover & Thomas)
  • Information Theory, Inference, and Learning Algorithms (MacKay)

Network Science

  • Networks, Crowds, and Markets (Easley & Kleinberg)
  • Social and Economic Networks (Jackson)

Markov Chain

  • Markov Chains (Norris)
  • Markov Chains and Mixing Times (Levin et al.)

Game Theory

  • Algorithmic Game Theory (Nisan et al.)
  • Multiagent Systems (Shoham & Leyton-Brown)

Combinatorics

  • The Probabilistic Method (Alon & Spencer)
  • A First Course in Combinatorial Optimization (Lee)

Algorithm

  • Introduction to Algorithms (Cormen et al.)
  • Randomized Algorithms (Motwani & Raghavan)
  • Approximation Algorithms (Vazirani)

Geometric View

  • Topological Data Analysis (Wasserman)
  • Methods of Information Geometry (Amari & Nagaoka)
  • Algebraic Geometry and Statistical Learning Theory (Watanabe)

Some Lectures

Math Backgrounds

I selected essential topics for machine learning.
Personally, I think more analysis / matrix / geometry never hurts.

Probability

  • Probability: Theory and Examples (Durrett)
  • Theoretical Statistics (Keener)
  • Stochastic Processes (Bass)
  • Probability and Statistics Cookbook (Vallentin)

Linear Algebra

  • Linear Algebra (Hoffman & Kunze)
  • Matrix Analysis (Horn & Johnson)
  • Matrix Computations (Golub & Van Loan)
  • The Matrix Cookbook (Petersen & Pedersen)

Large Deviations

  • Concentration Inequalities and Martingale Inequalities (Chung & Lu)
  • An Introduction to Matrix Concentration Inequalities (Tropp)

Blogs

About

Personal and biased selection of ML resources