Machine Learning for the Web
This is a repository for the "Machine Learning for the Web" class at ITP, NYU
Libraries like TensorFlow.js and ml5.js unlocked new opportunities for interactive machine learning projects in the browser. The goal of this class is to learn and understand common machine learning techniques and apply them to generate creative outputs in the browser.
This class will start with running pre-trained models and re-training models in the browser using high-level APIs from ml5.js, as well as explore the Layer APIs from TensorFlow.js to create models from scratch using custom data. This class will also cover preparing the dataset for training models.
At the completion of this course, students will have a better understanding of common and popular machine learning models, how do they work, how to train these models, and their use case to creative projects. The output of the class will be interactive ML web applications.
The topics that will be covered are Image/Sound/Doodle Classification, Face/Pose Recognition, Image Style Transfer, pix2pix Image Transformation, and Image Synthesis. The techniques and neural networks we will use and build are Transfer Learning, Convolutional Neural Network, Generative Adversarial Network and more.
Prospective students are expected to have taken an ICM (Introduction to Computational Media) course, or have equivalent programming experience with JavaScript, HTML, CSS.
Info
- Yining Shi, Wednesday, 6:30pm-9:00pm, Room 20, 3/27/2019 - 5/8/2019
- Office Hours
- All class dates
Requirements
- You are required to attend all class meetings and submit all weekly assignments.
- Grading (pass/fail) will be based on a combination of factors:
- Attendance, participation in class discussion, and engagement in other students' projects (25%)
- Quality of weekly assignments (50%)
- Last Assignment (25%)
Help
- How to push code to a Github Repo and host sketch on Github
- How to update your code and push it to Github
- Video: How to host p5 sketch on github pages
Get started
To run each examples, open your terminal, type in the following commands:
$ git clone https://github.com/yining1023/machine-learning-for-the-web.git
$ cd machine-learning-for-the-web
$ python -m SimpleHTTPServer # $ python3 -m http.server (if you are using python 3)
Go to localhost:8000 in your browser, you will see a directory list like this:
- week1-intro/
- week2-ImageClassifier-KNN-Posenet/
- week3-doodleclassifier/
- week4-styleTransfer/
- week5-pix2pix/
- week6-runway/
Click into each week, you will see the example for each week.
Syllabus
Week 1 Introduction to Machine Learning, MobileNet
-
Notes: Introduction to Machine Learning
-
Coding session:
- Installing ml5.js
- Running Image Classification example with ml5.js
- Installing tf.js
- Running examples with tf.js
Week 2 Image Classification (Transfer Learning, KNN Classifier, PoseNet)
Week 3 Train our own CNN model from scratch: Doodle Classifier
Week 4 Image Transformation (Part 1) Style transfer
-
Review: Showing the homework from last week, discuss any difficulties and other creative outputs
-
Coding session:
- Setup Spell.run training environment
- Training a new Style Transfer model
- Running Style Transfer model in ml5.js
Week 5 Image Transformation (Part 2) pix2pix, GAN
-
Review: Showing the homework from last week, discuss any difficulties when you are training your own style trasnfer model
-
Coding session:
- Running pix2pix with ml5.js
- Setup Spell.run training environment
- Prepare dataset for pix2pix
- Training a new pix2pix model
Week 6 RunwayML, Speech recognition, Sketch RNN, Body pix
-
More code examples and demo here
-
Review: Present your final assignment proposal and the experiments you did so far
-
Coding session:
- Runway ML
-
Homework: Work on the final assignment, prepare the presentation