Neuron Ablation in Model M
Andrea-de-Varda opened this issue · comments
My apologies if I bother you again. In your paper What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models you describe the results you obtained by ablating N% neurons in the model, and evaluating its performance (BLEU, perplexity) without a probe. However, I cannot find any documentation relative to that part. Are you planning to share it?
Thank you!
Thanks for reaching out! Do you mean the code for ablating in the machine translation model (OpenNMT-py) itself?
Yes, actually I wanted to do that with BERT, but that would be a good place to start
The ablation experiments in the paper were specifically run using code in https://github.com/dabbler0/seq2seq-attn-modify as far as I recall, but this was targeting a (now) really old lua version of seq2seq. We are working on bringing the ability to ablate/set neurons in transformer models, but haven't had the resources to finish the implementation yet unfortunately.
Is this something you might be potentially interested in implementing? We are happy to provide support and help out!
I would really love to help, but unfortunately I don't think I am capable of doing something like that, I am very sorry!
No worries, I understand. Keep an eye on the CHANGELOG (https://github.com/fdalvi/NeuroX/blob/master/CHANGELOG.md) for when we add it (hopefully soon in the coming months!)