Aniloid2 / EmpiricalPunctuationInsertionAttacks

Repository containing code for the paper: "Using Punctuation as an Adversarial Attack on Deep Learning Based NLP Systems: An Empirical Study"

Running EmpiricalPunctuationInsertionAttacks

Setup

Installation

Unzip TextAttack (please use this version as it processes punctuation differently)

unzip TextAttack

Conda

If conda isn't installed:

Get it from here:

https://www.anaconda.com/products/individual-d

If conda is installed:

Create env:

conda create --name PAA python=3.8.5 pip=20.2.4
source activate PAA

Please use the exact pip and python versions for initial testing, it's untested for other verisons.

Dependencies

THE FOLLOWING TWO COMMANDS SHOULD INSTALL ALL DEPENDENCIES AND TEXTATTACK FROM THIS REPO USING THE SETUP FILE (MAKE SURE YOU INSTALL THE VERSION OF TEXTATTACK IN THIS REPO)

pip install --use-feature=2020-resolver -r local_run.txt
pip install -e ./

All dependencies should be installed by now

Note, treie == CATE == PAA == EmpiricalPunctuationInsertionAttacks == EPIA (we changed the name last minute, we will update the name in the future)

Sentiment MR

In Classification_Tests/ run:

python classification_test.py --recipe 'punctuation_attack';

This will use the ' and - punctuation symbol to attack MR on bert

python classification_test.py --recipe 'pso_paa';

Will use paa as a multi-level attack together with SememePSO

To plot the mr results in Classification_Tests/ run:

python plot_classification_table.py

Entailment MNLI

For MNLI Entailment on BERT In Entailment_Test/ run:

python entailment.py --recipe 'punctuation_attack';
python plot_entialment_test.py

Question Answering

For Question Answering on BERT In Question_Answering_Test/

python question_answering.py --recipe 'punctuation_attack';
python plot_question_answering.py