VijayKalmath / AdversarialTraining_in_Distilation_Of_BERT

Implementation of Adversarial Training for BERT and BERT-Like Models and Analysis of effects of model compression on Robustness of a model

AdversarialTraining in Distillation OF BERT

In the recent years, transformer-based language learning models like BERT have been one of the most popular architectures used in the research related to natural lan- guage processing. Productionizing these models under constrained resources such as edge computing require smaller versions of BERT like DistilBERT. However, it is a concern that these models have inadequate robustness against adversarial examples and attacks. This paper evaluates the performance of various models built on the ideas of adversarial training and GAN BERT finetuned on SST-2 dataset. Further the experiments in this paper seek to find evidence on whether knowledege distillation preserves robustness in the student models.

Authors :

Vijay Kalmath Department of Data Science [e-mail]

Amrutha Varshini Sundar Department of Data Science [e-mail]

Sean Chen Department of Computer Science [e-mail]

Google Drive Link to Models : https://drive.google.com/drive/folders/1q1TGTOl6BftZzn1ZidzvN8GKVRbTdrjM?usp=sharing