juanmc2005 / MetricAMI

Official companion repository for the paper "A Metric Learning Approach to Misogyny Categorization" at the 5th Workshop on Representation Learning for NLP, ACL 2020

A Metric Learning Approach to Misogyny Categorization

This is the official companion repository for the paper, published at the 5th Workshop on Representation Learning for NLP (RepL4NLP) at ACL 2020.

Access to the full paper

The project contains all the code needed to run the experiments.

Installation

We recommend you first install PyTorch as described in the official website according to your hardware capabilities, making sure the version is at least 1.4.0 (ideally you should install exactly this version to avoid any compatibility issues).

1. Clone the repository and cd MetricAMI
2. Run pip install -r requirements.txt

Prepare the Dataset

1. Obtain the official Evalita 2018 AMI dataset from here
2. Run the preparation script to pre-tokenize the data and split the training set into train and dev:

python prepare.py --path <AMI_PATH> --out <OUTPUT_PATH>

where <AMI_PATH> is the directory containing en_training.tsv and en_testing.tsv, and <OUTPUT_PATH> any directory to put the preprocessed data with en as the last directory.

Notice that the vocabulary will also be generated by this script.

At this point you should have a directory tree like this:

- <NEW_AMI_PATH>/
  - en/
    - train/
    - dev/
    - test/
    - ami_vocab.txt

Train an LSTM Model

To train this model you will need the generated vocabulary as well as a GENSIM KeyedVectors dump of CBOW Wikipedia word embeddings (included in this repo). You can of course train any of the other loss functions in a similar manner by modifying the parameters (see here for the parameter list and description).

python train.py \
       --path <NEW_AMI_PATH> \
       --vocab /path/to/ami_vocab.txt \
       --word2vec-model word_embeddings/ami2vec.kv \
       --epochs 60 --save --batch-size 32 --log-interval 50 \
       --model lstm --loss softmax --lr 1e-3 \
       --exp-path <EXPERIMENT_PATH>

This model will need around 700Mb of GPU RAM.

Train a BERT Model

Training BERT is easier thanks to the Huggingface library. As with the LSTM, any other loss function from the paper can also be trained (see here).

python train.py \
       --path <NEW_AMI_PATH> \
       --epochs 60 --save --batch-size 32 --log-interval 50 \
       --model bert --loss softmax --lr 1e-5 \
       --exp-path <EXPERIMENT_PATH>

BERT will need around 4.5Gb of GPU RAM.

Results

Our results show that none of the considered losses can outperform the regular cross entropy, and we outperform the Evalita 2018 winner with our fine-tuned BERT model.

A detailed analysis of the results can be found in the paper.

Citation

If our work has been useful to you, please cite our paper:

@inproceedings{coria-etal-2020-metric,
    title = "A Metric Learning Approach to Misogyny Categorization",
    author = "Coria, Juan Manuel  and
      Ghannay, Sahar  and
      Rosset, Sophie  and
      Bredin, Herv{\'e}",
    booktitle = "Proceedings of the 5th Workshop on Representation Learning for NLP",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.repl4nlp-1.12",
    pages = "89--94"
}