Official coursebook information
Lectures:
Fri 13:15-15:00 in CO2
Exercises:
Fri 15:15-17:00 in BC01, Zoom
This course teaches an overview of modern mathematical optimization methods, for applications in machine learning and data science. In particular, scalability of algorithms to large datasets will be discussed in theory and in implementation.
- Instructors:
- Martin Jaggi martin.jaggi@epfl.ch
- Nicolas Flammarion nicolas.flammarion@epfl.ch
- Assistants:
- Anastasia Koloskova anastasia.koloskova@epfl.ch
- Sai Praneeth Karimireddy sai.karimireddy@epfl.ch
- Thijs Vogels thijs.vogels@epfl.ch
Contents:
Convexity, Gradient Methods, Proximal algorithms, Subgradient Methods, Stochastic and Online Variants of mentioned methods, Coordinate Descent, Frank-Wolfe, Accelerated Methods, Primal-Dual context and certificates, Lagrange and Fenchel Duality, Second-Order Methods including Quasi-Newton Methods, Derivative-Free Optimization.
Advanced Contents:
Parallel and Distributed Optimization Algorithms
Computational Trade-Offs (Time vs Data vs Accuracy), Lower Bounds
Non-Convex Optimization: Convergence to Critical Points, Alternating minimization, Neural network training
Nr | Date | Topic | Materials | Exercises |
---|---|---|---|---|
#1 | 21.2. | Introduction, Convexity | notes, slides | lab01 |
#2 | 28.2. | Gradient Descent | notes, slides | lab02 |
#3 | 6.3. | Projected Gradient Descent | notes, slides | lab03 |
#4 | 13.3. | Proximal and Subgradient Descent | notes, slides | lab04 |
#5 | 20.3. | Stochastic Gradient Descent, Non-Convex Optimization | notes, slides | lab05 |
#6 | 27.3. | Non-Convex Optimization, Accelerated Gradient Descent | notes, slides | lab06 |
#7 | 3.4. | Newton's Method & Quasi-Newton | notes, slides | lab07 |
. | 10.4. | easter vacation |
- | |
. | 17.4. | easter vacation |
- | |
#8 | 24.4. | Frank-Wolfe | notes, slides | lab08 |
#9 | 1.5. | Coordinate Descent | notes, slides | lab09 |
#10 | 8.5. | Duality, Gradient-free, adaptive methods | notes, slides | lab10 |
#11 | 15.5. | Opt for ML in Practice | notes, slides | Q&A |
#12 | 22.5. | Opt for ML in Practice | notes, slides | Q&A Projects |
#13 | 29.5. | Mini-Project week |
The weekly exercises consist of a mix of theoretical and practical Python
exercises for the corresponding topic each week (starting week 2). Solutions to theory exercises are available here, and for practicals in the lab folder.
A mini-project
will focus on the practical implementation: Here we encourage students to investigate the real-world performance of one of the studied optimization algorithms or variants, helping to provide solid empirical evidence for some behaviour aspects on a real machine-learning task. The project is mandatory and done in groups of 3 students. It will count 20% to the final grade. Project reports (3 page PDF) are due June 12th. Here is a detailed project description.
Final written exam in exam session in August, Tuesday 11.08.2020 from 08h15 to 11h15 in the SwissTech Convention Center. Format: Closed book. Theoretical questions similar to exercises. You are allowed to bring one cheat sheet (A4 size paper, both sides can be used), either handwritten or 11 point minimum font size. For practice: exam 2019, solutions 2019, exam 2018.
- Convex Optimization: Algorithms and Complexity, by SĂ©bastien Bubeck (free online)
- Convex Optimization, Stephen Boyd and Lieven Vandenberghe (free online)
- Introductory Lectures on Convex Optimization, Yurii Nesterov (free online)