refracta / TIE-KD

TIE-KD

Introduction

we introduce a novel Teacher-Independent Explainable Knowledge Distillation (TIE-KD) framework that streamlines the knowledge transfer from complex teacher models to compact student networks, eliminating the need for architectural similarity

Related paper

TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation

Installation

Please refer to get_started.md (html) for installation and dataset_prepare.md (html) for dataset preparation.

Get Started

We provide train.md (html) and inference.md (html) for the usage of this toolbox.

Results and models

Teachers

Model	Backbone	Train Epoch	Abs Rel	RMSE	Config	Download
Adabins	EfficientNetB5-AP	24	0.0593	2.3309	config	model
BTS	ResNet-50	24	0.0586	2.4798	config	model
Depthformer	SwinL-w7-22k	24	0.0513	2.1038	config	model

Students

Teacher	Method	Loss	Backbone	Train Epoch	Abs Rel	RMSE	Config	Download
-	Baseline	SI	MobileNetV2	24	0.0663	2.5625	config	model
-	Baseline	SI	ResNet18	24	0.0634	2.5311	config	model
-	Baseline	SI	ResNet50	24	0.0605	2.4159	config	model
Adabins	Res-KD	SSIM	MobileNetV2	24	0.0697	2.5639	config	model
Adabins	Res-KD	MSE	MobileNetV2	24	0.0786	2.6964	config	model
Adabins	Res-KD	SI	MobileNetV2	24	0.0739	2.7371	config	model
Adabins	Res-KD	SSIM, SI	MobileNetV2	24	0.0701	2.5833	config	model
Adabins	Res-KD	SSIM, MSE	MobileNetV2	24	0.0808	2.6943	config	model
Adabins	TIE-KD	L_DPM	MobileNetV2	24	0.0718	2.5433	config	model
Adabins	TIE-KD	L_DEPTH	MobileNetV2	24	0.0696	2.4646	config	model
Adabins	TIE-KD	L_DPM, L_DEPTH	MobileNetV2	24	0.0654	2.4315	config	model
BTS	Res-KD	SSIM	MobileNetV2	24	0.0697	2.6357	config	model
BTS	Res-KD	MSE	MobileNetV2	24	0.0820	2.7440	config	model
BTS	Res-KD	SI	MobileNetV2	24	0.0782	2.8106	config	model
BTS	Res-KD	SSIM, SI	MobileNetV2	24	0.0690	2.6168	config	model
BTS	Res-KD	SSIM, MSE	MobileNetV2	24	0.0914	2.7983	config	model
BTS	TIE-KD	L_DPM	MobileNetV2	24	0.0722	2.6459	config	model
BTS	TIE-KD	L_DEPTH	MobileNetV2	24	0.0679	2.5694	config	model
BTS	TIE-KD	L_DPM, L_DEPTH	MobileNetV2	24	0.0656	2.4984	config	model
Depthformer	Res-KD	SSIM	MobileNetV2	24	0.0692	2.5009	config	model
Depthformer	Res-KD	MSE	MobileNetV2	24	0.0805	2.6029	config	model
Depthformer	Res-KD	SI	MobileNetV2	24	0.0724	2.6717	config	model
Depthformer	Res-KD	SSIM, SI	MobileNetV2	24	0.0682	2.5709	config	model
Depthformer	Res-KD	SSIM, MSE	MobileNetV2	24	0.0770	2.6391	config	model
Depthformer	TIE-KD	L_DPM	MobileNetV2	24	0.0713	2.5241	config	model
Depthformer	TIE-KD	L_DEPTH	MobileNetV2	24	0.0698	2.4805	config	model
Depthformer	TIE-KD	L_DPM, L_DEPTH	MobileNetV2	24	0.0657	2.4402	config	model

Teacher	Method	Loss	Backbone	Train Epoch	Abs Rel	RMSE	Config	Download
Adabins	TIE-KD	L_DPM, L_DEPTH	ResNet18	24	0.0628	2.4029	config	model
Adabins	TIE-KD	L_DPM, L_DEPTH	ResNet50	24	0.0597	2.3060	config	model
BTS	TIE-KD	L_DPM, L_DEPTH	ResNet18	24	0.0635	2.4527	config	model
BTS	TIE-KD	L_DPM, L_DEPTH	ResNet50	24	0.0615	2.4019	config	model
Depthformer	TIE-KD	L_DPM, L_DEPTH	ResNet18	24	0.0624	2.3963	config	model
Depthformer	TIE-KD	L_DPM, L_DEPTH	ResNet50	24	0.0586	2.2821	config	model

Viewer

We provide a viewer that can simultaneously check the results of different methods for three models.

• Move between images with '←' and '→' keys
• Display diff of other images for that image with 1, 2, 3, 4, 5, 6, 7, 8, 9 numeric keys (press the same button again to toggle back)

|AdaBins| |BTS| |Depthformer|

Acknowledgement

This repo benefits from Monocular-Depth-Estimation-Toolbox. Please also consider citing them.

Special thanks

@refracta - Developing a viewer to check the results, Qualitative and quantitative comparison of data for data selection.