zhaowangji / knowledge-distillation-for-unet

An implementation of Knowledge distillation for segmentation, to train a small (student) UNet from a larger (teacher) UNet thereby reducing the size of the network while achieving performance similar to the heavier model.

Knowledge Distillation for UNet

An implementation of Knowledge distillation for segmentation, to train a small (student) UNet from a larger (teacher) UNet thereby reducing the size of the network while achieving performance similar to the heavier model.

Results:

Dataset: Carvana Image Masking Challenge

Models trained without knowledge distillation

Models trained with knowledge distillation

References

• Distilling the Knowledge in a Neural Network - Geoffrey Hinton, Oriol Vinyals, Jeff Dean

• Structured Knowledge Distillation for Dense Prediction - Yifan Liu, Changyong Shun, Jingdong Wang, Chunhua Shen

• On Compressing U-net Using Knowledge Distillation - Karttikeya Mangalam, Mathieu Salzamann