Giters

tianyic / only_train_once_personal_footprint

OTOv1-v3, NeurIPS, ICLR, TMLR, DNN Training, Compression, Structured Pruning, Erasing Operators, CNN, Diffusion, LLM

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Understanding pruning_steps

iamanigeeit opened this issue · comments

commented

Hello @tianyic ,

I see that pruning_steps determines how many steps we take before pruning:
https://github.com/tianyic/only_train_once/blob/7e930f6ae6cab71659fb921671f6f3921828d7c4/only_train_once/optimizer/hesso.py#L29

  1. Is it correct to say that if pruning_periods = 1, then pruning_steps = how many steps before we prune once? If it makes more sense to prune after every epoch, i will set pruning_steps = steps_per_epoch.

  2. What does pruning_periods do?

Thanks in advance!

commented

@iamanigeeit

  1. Yes, your understanding is correct. It makes sense to set pruning_steps = steps_per_epoch .
  2. pruning_periods controls how many groups to be pruned incrementally. For example, if target_group_sparsity=0.7, pruning_periods=10, pruning_steps=200, then every pruning_steps /pruning_periods =20 steps will yield target_group_sparsity / pruning_periods =0.07 groups as redundant and project them onto zero.