fasterai

FasterAI: A repository for making smaller and faster models with the FastAI library.

Fasterai now exists here for fastai 2.0 !

It allows you to use techniques such as:

Knowledge Distillation
Pruning
Batch Normalization Folding
Matrix Decomposition

Usage:

Knowledge Distillation

Used as a callback to make the student model train on soft-labels generated by a teacher model.

 KnowledgeDistillation(student:Learner, teacher:Learner)
You only need to give to the callback function your student learner and your teacher learner. Behind the scenes, fasterai will take care of making your model train using knowledge distillation

Sparsify the network

Used as a callback, will iteratively replace the lowest-norm parameters by zeroes. More information in this blog post

SparsifyCallback(learn, sparsity, granularity, method, criteria, sched_func)
sparsity: the percentage of sparsity that you want in your network

granularity: on what granularity you want the sparsification to be operated (currently supported: weight, kernel, filter)

method: either local or global, will affect the selection of parameters to be choosen in each layer independently (local) or on the whole network (global).

criteria: the criteria used to select which parameters to remove (currently supported: l1, grad)

sched_func: which schedule you want to follow for the sparsification (currently supported: any scheduling function of fastai, i.e annealing_linear, annealing_cos, ... and annealing_gradual, the schedule proposed by Zhu & Gupta) (shown in Figure below)

Prune the network

Will physically remove the parameters zeroed out in step before. More information in this blog post

Warning: this only works when filter sparsifying has been performed and for fully feed-forward architectures such as VGG16.

pruner = Pruner()
pruned_model = pruner.prune_model(learn.model)
You just need to pass the model whose filters has previously been sparsified and FasterAI will take care of removing them.

Batch Normalization Folding

Will remove batch normalization layers by injecting its normalization statistics (mean and variance) into the previous convolutional layer. More information in this blog post

bn_folder = BN_Folder()
bn_folder.fold(learn.model))
Again, you only need to pass your model and FasterAI takes care of the rest. For models built using the nn.Sequential, you don't need to change anything. For others, if you want to see speedup and compression, you actually need to subclass your model to remove the batch norm from the parameters and from the forward method of your network.

Fully-Connected Layers Decomposition

Will replace fully-connected layers by a factorized version that is more parameter efficient.

FCD = FCDecomposer()
decomposed_model = FCD.decompose(model, percent_removed)
The percent_removed corresponds to the percentage of singular values removed (k value above).

nathanhubens / fasterai1