This repository is a Re-implementation for creating the benchmarks in the paper Modifying Memories of Transformer Models. Due to randomness, the selected subset of facts for modification are different from the original paper.
To generate modified facts:
conda create -n lama37 -y python=3.7 && conda activate lama37
pip install -r requirements.txt
python -m spacy download en
mkdir lama_data && cd lama_data
wget https://dl.fbaipublicfiles.com/LAMA/data.zip
unzip data.zip
rm data.zip
cd ..For a full list of available models, please refer to the LAMA repo.
bash ./download_bert_base.sh
python create_trex.py
By default, a dict mapping uuids to the modified answers are stored in modification/change_list_{num_modified_facts}.json. You can use this dict to finetune the model on the supporting facts on TREx and test it on the corresponding LAMA probes.
For example, the answer of the fact with uuid 6904327c-5211-4edb-ad8c-bcefee2149c6 (in lama_data/data/TREx/P276.jsonl) is modified from Atlanta to Ukraine. You could use all the masked sub_surface sentences in P276.jsonl for finetuning the model, and the template [X] is located in [Y]. for predicate P276 to generate the masked probing sentence for test. These templates are given in lama_data/data/relations.jsonl if you followed our Step 2.
The zsRE implementation is based on KILT.
mkdir zsre_data && cd zsre_data
wget http://dl.fbaipublicfiles.com/KILT/structured_zeroshot-train-kilt.jsonl
cd .. Since a finetuned model can easily achieve perfect test accuracy in our setting, we simply choose any fact with at least one eval question for modification.
The examples are given in modification.zip. We give an example subset of 32 modified facts for TREx in modification/change_list_32.json.
For zsRE, we give an example train/eval split (modification/zsre_train_unmodified.jsonl and modification/zsre_eval_unmodified.jsonl), and a subset of 32 modified facts (modification/zsre_train_modified_32.jsonl and modification/zsre_eval_modified_32.jsonl).