pynchmeister / fromthetensor

From the Tensor to Stable Diffusion, a rough outline for a 9 week course.

From the Tensor to Stable Diffusion

Inspired by From the Transistor.

Machine learning is hard, a lot of tutorials are hard to follow, and it's hard to understand software 2.0 from first principles.

You wanna be an ML engineer? Well, here's the steps to get good at that:

1. Download a paper
2. Implement it
3. Keep doing this until you have skills

-- George Hotz

Section 1: Intro: Cheating our way past the Tensor -- 1 week

• So about those Tensors -- Course overview. Describe how Deep Learning models are buildable using Tensors, and how different architectures like CNNs and RNNs use Tensors in different ways. Understand the concept of backpropagation and gradient descent. [video]

• Accelerated learning -- Training on a personal computer may limit the reach of this course. Using something like Google Colab will allow anyone with a computer to play.

Section 2: Deep Learning: What is deep learning anyway? -- 1 week

• Building a simple Neural Network -- Your first little program! Getting the model working and learning the basics of deep learning. [code] [video]

• Building a simple CNN -- An intro chapter to deep learning, learn how to build a simple CNN and understand the concepts of convolution and pooling. [code] [video]

• Building a simple RNN -- Learn the basics of Recurrent Neural Networks and understand the concept of "memory" that helps them store states of previous inputs. [code] [video]

Section 3: Implementing Papers (Part 1): Vision models -- 3 weeks

• Implementing LeNet -- Learn about the LeNet architecture and its application. [code] [paper]

• Implementing AlexNet -- Learn how to implement AlexNet for image classification tasks. [code] [paper]

• Implementing ResNet -- Learn how to implement ResNet for image classification tasks. [code] [paper]

• Building a DCGAN -- Learn how to build a DCGAN and the concept of adversarial training. [code] [paper]

Section 4: Implementing Papers (Part 2): Language models -- 3 weeks

• Implementing GRU and LSTM -- Learn about the concepts of LSTM and GRU cells. [code] [paper]

• Implementing CBOW and Skip-Gram -- Learn about the word2vec architecture and its application. [code] [paper]

• Building a Transformer -- Learn about the transformer architecture and its application. [code] [paper]

• Fine-tuning a BERT -- Learn about the BERT architecture and fine-tuning a pre-trained model. [code] [paper]

Section 5: Implementing Papers (Part 3): Vision-Language models -- 1 week

• Building a Stable Diffusion model -- Learn about the Stable Diffusion architecture and its application in image generation tasks. [code] [paper]