The course is devoted to modern generative models (mostly in the application to computer vision).
We will study the following types of generative models:
- autoregressive models,
- latent variable models,
- normalization flow models,
- adversarial models,
- diffusion models.
Special attention is paid to the properties of various classes of generative models, their interrelationships, theoretical prerequisites and methods of quality assessment.
The aim of the course is to introduce the student to widely used advanced methods of deep learning.
The course is accompanied by practical tasks that allow you to understand the principles of the considered models.
| # | Date | Description | Slides |
|---|---|---|---|
| 0 | September, 12 | Logistics and intro. | slides |
| 1 | September, 12 | Lecture: Generative models overview and motivation. Problem statement. Divergence minimization framework. Autoregressive modelling. | slides |
| Seminar: Introduction. Maximum likelihood estimation. Kernel density estimation. Histograms and KDE. | notebook | ||
| 2 | September, 19 | Lecture: Autoregressive models (WaveNet, PixelCNN). Bayesian Framework. Latent Variable Models (LVM). Variational lower bound (ELBO). | slides |
| Seminar: MADE theory and practice. PixelCNN implementation hints. | notebook | ||
| 3 | September, 26 | Lecture: EM-algorithm, amortized inference. ELBO gradients, reparametrization trick. Variational Autoencoder (VAE). | slides |
| Seminar: CNNs. Additional notes on autoregressive models. Latent Variable Models. | notebook CNNs_note |
||
| 4 | October, 3 | Lecture: VAE limitations. Posterior collapse and decoder weakening. Tighter ELBO (IWAE). Normalizing flows prerequisities. | slides |
| Seminar: Gaussian Mixture Models (GMM). MLE for GMM, ELBO and EM-algorithm for GMM. VAE basics. | notebook | ||
| 5 | October, 10 | Lecture: Normalizing Flow (NF) intuition and definition. Forward and reverse KL divergence for NF. Linear flows. | slides |
| Seminar: VAE implementation hints. IWAE model. | notebook | ||
| 6 | October, 17 | Lecture: Autoregressive flows (gausian AR NF/inverse gaussian AR NF). Coupling layer (RealNVP). NF as VAE model. | slides |
| Seminar: Flows. Planar flows. Forward KL vs Reverse KL. Planar flows via Forward KL and Reverse KL. | notebook planar_flow_practice autograd_jacobian |
||
| 7 | October, 24 | Lecture: 7. Discrete data vs continuous model. Model discretization (PixelCNN++). Data dequantization: uniform and variational (Flow++). ELBO surgery and optimal VAE prior. Flow-based VAE prior. | slides |
| Seminar: RealNVP hints. Discretization of continuous distribution (MADE++). | notebook | ||
| 8 | October, 31 | Lecture: 8. Flows-based VAE posterior vs flow-based VAE prior. Likelihood-free learning. GAN optimality theorem. | slides |
| Seminar: VAE with learnable prior. Aggregated posterior. Integer discrete flows. | notebook | ||
| 9 | November, 7 | Lecture: Vanishing gradients and mode collapse, KL vs JS divergences. Adversarial Variational Bayes. Wasserstein distance. Wasserstein GAN (WGAN). | slides |
| Seminar: KL vs JS divergences. Mode collapse. Vanilla GAN in 1D coding. | notebook notebook_done |
||
| 10 | November, 14 | Lecture: WGAN with gradient penalty (WGAN-GP). Spectral Normalization GAN (SNGAN). f-divergence minimization. GAN evaluation. | slides |
| Seminar: 1-Wasserstein distance introduction: discrete and continuous case. WGAN theory. Vanilla GAN on 2D data practice. | notebook WGAN_theory |
||
| 11 | November, 21 | Lecture: GAN evaluation (Inception score, FID, Precision-Recall, truncation trick). Discrete VAE latent representations. | slides |
| Seminar: WGANs on multimodal 2D data. GANs zoo. Evolution of GANs. StyleGAN implementation: start discussion. | notebook GANevolution StyleGAN_not_completed |
||
| 12 | November, 28 | Lecture: Vector quantization, straight-through gradient estimation (VQ-VAE). Gumbel-softmax trick (DALL-E). Neural ODE. Adjoint method. | slides |
| Seminar: VQ-VAE implementation hints. StyleGAN coding and assessing. | notebook StyleGAN |
||
| 13 | December, 5 | Lecture: Continuous-in-time NF (FFJORD, Hutchinson's trace estimator). Kolmogorov-Fokker-Planck equation and Langevin dynamic. SDE basics. Score matching. | slides |
| Seminar: CNF theory. Langevin Dynamics. Score matching practice. | notebook | ||
| 14 | December, 12 | Lecture: Noise conditioned score network (NCSN). Gaussian diffusion process. Denoising diffusion probabilistic model (DDPM). | slides |
| Seminar: NCSN and DDPM: theory and implementation on 2D data. | notebook | ||
| December, 19 | Oral exam |
| Homework | Date | Deadline | Description | Link |
|---|---|---|---|---|
| 1 | September, 22 | October, 6 |
|
  |
| 2 | October, 6 | October, 20 |
|
|
| 3 | October, 20 | November, 3 |
|
|
| 4 | November, 3 | November, 17 |
|
|
| 5 | November, 17 | December, 1 |
|
|
| 6 | December, 1 | December, 15 |
|
|
- 6 homeworks each of 13 points = 78 points
- oral cozy exam = 26 points
- maximum points: 78 + 26 = 104 points
- telegram: @roman_isachenko
- e-mail: roman.isachenko@phystech.edu