LAMA: LAnguage Model Analysis
This repository is forked from https://github.com/facebookresearch/LAMA
LAMA is a probe for analyzing the factual and commonsense knowledge contained in pretrained language models.
https://dl.fbaipublicfiles.com/LAMA/data.zip
The dataset for the LAMA probe is available at LAMA contains a set of connectors to pretrained language models.
LAMA exposes a transparent and unique interface to use:
- Transformer-XL (Dai et al., 2019)
- BERT (Devlin et al., 2018)
- ELMo (Peters et al., 2018)
- GPT (Radford et al., 2018)
- RoBERTa (Liu et al., 2019)
Actually, LAMA is also a beautiful animal.
The LAMA probe
To reproduce the results:
1. Create conda environment and install requirements
Use a separate conda environment. It can be created by running:
conda create -n lama37 -y python=3.7 && conda activate lama37
pip install -r requirements.txt
2. Download the data
wget https://dl.fbaipublicfiles.com/LAMA/data.zip
unzip data.zip
rm data.zip
3. Download the models
Disk usage: ~55 GB.
Install spacy model
python3 -m spacy download en
Download the models
chmod +x download_models.sh
./download_models.sh
The script will create and populate a pre-trained_language_models folder. If you are interested in a particular model please edit the script.
4. Run the experiments
chmod +x run_experiment/*.sh
Run cased model
./run_experiment/run_experiment_cased.sh
Run uncased model
./run_experiment/run_experiment_uncased.sh
results will be logged in output/ and run_experiment/*.log.
Other versions of LAMA
LAMA-UHN
This repository also provides a script (scripts/create_lama_uhn.py
) to create the data used in (Poerner et al., 2019).
Negated-LAMA
This repository also gives the option to evalute how pretrained language models handle negated probes (Kassner et al., 2019), set the flag use_negated_probes
in scripts/run_experiments.py
. Also, you should use this version of the LAMA probe https://dl.fbaipublicfiles.com/LAMA/negated_data.tar.gz
Reference:
The LAMA probe is described in the following paper:
@inproceedings{petroni2019language,
title={Language Models as Knowledge Bases?},
author={F. Petroni, T. Rockt{\"{a}}schel, A. H. Miller, P. Lewis, A. Bakhtin, Y. Wu and S. Riedel},
booktitle={In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019},
year={2019}
}
Other References
-
(Kassner et al., 2019) Nora Kassner, Hinrich Schütze. Negated LAMA: Birds cannot fly. arXiv preprint arXiv:1911.03343, 2019.
-
(Poerner et al., 2019) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA. arXiv preprint arXiv:1911.03681, 2019.
-
(Dai et al., 2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdi. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860.
-
(Peters et al., 2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. NAACL-HLT 2018
-
(Devlin et al., 2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
-
(Radford et al., 2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
-
(Liu et al., 2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Licence
LAMA is licensed under the CC-BY-NC 4.0 license. The text of the license can be found here.