Clarification - applying the mask

[itay hubara]

Thank you for your interesting paper!
I went over your code and I tried to understand where do you apply the mask. My main interest was using your "Iterative Synaptic Flow Pruning" algorithm. Although I see the function in pruner class (line 52 pruners.py) I can't find where do you use it. For instance, in prune_loop, I would expect (at least for SynFlow) to see after pruner.mask() pruner.apply_mask (prune.py line 20). Also, I would expect in train.py after optimizer.step() (line 16) to see some parameter masking.
Surprisingly for the theory experiment, I do see that you apply the mask (schedule_conservation.py line 92) before calculating the next score.
Can you please help me sort this out.
Thanks

[Daniel Kunin]

Hi, we created custom dense and convolution layers where we added a binary mask that is applied during the forward pass (and then also during backward pass by autograd). In prune.py we call prune.score which applies the pruner's score function and then prune.mask which updates the mask according to the scores and sparsity. Because our layers always apply the mask we don't need to directly apply_mask to the weights. Hope that helps!

[Daniel Kunin]

Also to clarify, we recently removed the "apply_mask" call in the prune loop so we could accurately do iterative versions of SNIP. This call was redundant and means you loose information in the pruned weights. We couldn't call this for iterative versions of SNIP where pruned weights at one iteration can actually re-enter (this doesn't happen for SynFlow). But this is why the apply_mask call is still in the theory experiments because we didn't update them. Sorry for the confusion.

[itay hubara]

Thanks for the prompt reply! all make sense now