tinganchen / dominos

[NTU 2020 Master Thesis] Dominance in an Overall Sight: Attentive Channel Pruning via Filter Importance Learning

Dominance in an Overall Sight: Attentive Channel Pruning via Filter Importance Learning (DOMINOS)

Pruning, Model Compression, Efficient Inference, Neural Networks (PDF)

Requirements

• python3
• pytorch==1.7.1
• cudatoolkit==11.0.221
• numpy==1.19.2
• tensorboardx==1.4
• ptflops (Github)

Pretrained weights download

• CIFAR-10

  • VGG-16
  • ResNet-56
  • ResNet-110
  • DenseNet-40

• CIFAR-100

  • ResNet-56
  • ResNet-110

• SVHN

  • VGG-16
  • MobileNet-v2

• ImageNet

  • ResNet-50

Implementation

e.g. ResNet-56 on CIFAR-10.

Preparation of pretrained weights

cd dominos/resnet-56-cifar-10
mkdir pretrained/

Download the pretrained weights under the path dominos/resnet-56-cifar-10/pretrained/.

Dominant Structure Search (DSS) stage

python3 main.py --job_dir <pruning_results_dir> --teacher_dir <pretrain_weights_dir> --teacher_file <pretrain_weights_file> --refine None --arch resnet --teacher_model resnet_56 --student_model resnet_56_sparse --num_epochs 100 --train_batch_size 128 --eval_batch_size 100 --lr 0.01 --momentum 0.9 --miu 1 --sigma 0.2 --mask 0.3 --sparse_lambda 0.001 --sparse_lambda2 0.01 --lr_decay_step 30 --mask_step 200 --weight_decay 0.0002 --t 2 --thres 0.2

Fine-tuning stage

python3 ft.py --job_dir <finetuning_results_dir> --refine <pruning_results_dir> --num_epochs 100 --lr 0.05