ryancheunggit / tensorflow2_model_zoo

explore tensorflow 2

Tensorflow 2.0 playground

This is the repo for working with nerual networks with Tensorflow 2.0 beta. Tensorflow 2.0 is out now~

CPU only version installation:

pip install tensorflow

GPU version please consult GPU Guide

Contents

Stand alone examples

1. dataset == MNIST
   • subclass model with eager execution mode in custom loop
   • above + @tf.function decorator
   • above with tf.keras.Sequential model
   • above with tf.keras .fit api
   • metric learning with additive angular margin loss
     Reference: ArcFace: Additive Angular Margin Loss for Deep Face Recognition
2. dataset == cifar10
   • simple cnn model
   • above with mixup
     Reference: mixup: Beyond empirical risk minimization
   • above with ict
     Reference: Interpolation Consistency Training for Semi-Supervised Learning
3. dataset == titanic
   • unsupervised categorical feature embedding

Tutorial Notebooks - I call this 'Learn Tensorflow 2.0 the Hard Way'

1. Tensors Variables Operations and AutoDiff
2. AutoGraph
3. Custom Model and Layer
4. Optimizers
5. Loss Function

Losses

Loss	Reference	Year
ApproxNDCG	A General Approximation Framework for Direct Optimization of Information Retrieval Measures	2008
Smooth L1 Loss	Fast R-CNN	2015
ArcFace	ArcFace: Additive Angular Margin Loss for Deep Face Recognition	2018

Optimizers

Optimizer	Reference	Year
LARS	Large Batch Training of Convolutional Networks	2017
SWA	Averaging Weights Leads to Wider Optima and Better Generalization	2018
Yogi	Adaptive Methods for Nonconvex Optimization	2018
RAdam	On the Variance of the Adaptive Learning Rate and Beyond	2019
LAMB	Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019
Lookahead	Lookahead Optimizer: k steps forward, 1 step back	2019

Convolutional Neural Networks

Model	Reference	Year
AlexNet	ImageNet Classification with Deep Convolutional Neural Networks	2012
VGG	Very Deep Convolutional Networks for Large-Scale Image Recognition	2014
GoogleNet	Going Deeper with Convolutions	2015
ResNet	Deep Residual Learning for Image Recognition	2016
WideResNet	Wide Residual Networks	2016
SqueezeNet	SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size	2016
DenseNet	Densely Connected Convolutional Networks	2017
ResNeXt	Aggregated Residual Transformations for Deep Neural Networks	2017
SEResNeXt	Squeeze-and-Excitation Networks	2018
MobileNetV2	MobileNetV2: Inverted Residuals and Linear Bottlenecks	2018
ShuffleNetV2	ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design	2018
MnasNet	MnasNet: Platform-Aware Neural Architecture Search for Mobile	2019

Sequence Models

Model	Reference	Year
Transformer	Attention Is All You Need	2017

Graph Neural Networks

Model	Reference	Year
MPNN	Neural Message Passing for Quantum Chemistry	2017

Factorization Machines

See the benchmark result on how to use the benchmark runner to train these models on criteo dataset.

Model	Reference	Year
FM	Factorization Machines	2010
FFM	Field-aware factorization machines for CTR prediction	2016
FNN	Deep Learning over Multi-field Categorical Data: A Case Study on User Response Prediction	2016
AFM	Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks	2017
DeepFM	DeepFM: A Factorization-Machine based Neural Network for CTR Prediction	2017
NFM	Nerual Factorization Machines for Sparse Predictive Analytics	2017
xDeepFM	xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems	2018
AutoInt	AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks	2018
FNFM	Field-aware Neural Factorization Machine for Click-Through Rate Prediction	2019

Bag of Tricks

• Learning rate finder - Train the model over small number of iterations with increasing learning rates. Plot the loss values against the learning rates to identify sweet spot. Reference
• Monte Carlo Dropout - Kind of TTA procedure, that averages the predictions from model with dropout activated in inference mode. Reference