ICLR2020
Virtual ICLR 2020: https://iclr.cc/virtual
Asynchronous coverage!
Yann LeCun - The Future is Self-Supervised:
https://iclr.cc/virtual/calendar.html#tab-Thursday
Self-supervised learning: fill in the blanks: multimodal prediction e.g. for video it would be missing frames, missing parts of frames etc. Capture
Feed-forward NN has one deterministic output. What about complex outputs? Inference through constraint relaxation. Use energy- based models. cf factor graphs in graphical models.
Uncertainty for YLC here: many y are compatible with an x.
How to handle multiple outputs - latent variables.
Collobert-Weston Denoising Autoencoder is BERT-esque: "Natural Language Processing (Almost) from Scratch" (http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf)
Siamese Networks/Constrastive Embeddings: show similar and different pairs of images. High comp cost. Many ways for two images to be different!
CF. GANs.
Link to sparse coding.
Temporal reg. Temporal Invariance.
Learning forward models of the world for control. Train a policy network to produce an action from the state. Cf model predictive control.
Text: discrete. Video - high-dim continuous variables.
Bet: reg. latent-variables energy-based models.
There is no thing as AGI. Even Human Intelligence is narrow.
Bengio - Deep Learning Priors Associated with Conscious Processing
https://iclr.cc/virtual/calendar.html#tab-Thursday
Stopped at 22 mins