salesforce / adversarial-polyglots-data

Adversarial Polyglots (data)

This repository is a companion repository to adversarial-polyglots and contains data for the paper "Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots" (NAACL-HLT 2021).

Authors: Samson Tan and Shafiq Joty

XNLI_CleanDL test set

The XNLI_CleanDL used in our CAT experiments can be found under XNLI_cleanDL. In this setting, the premise and hypothesis are randomly chosen from different languages in the original 15 languages. This setting prevents the NLP model from using lexical overlap as a shortcut to doing well on the NLI task.

If you wish to generate variations of this test set, you can use this script from the code repository.

Translated XNLI Data

For the research community's convenience, we release the XNLI test set we translated to 18 other languages using machine-translation models released by Helsinki-NLP under their OPUS-MT project.

Languages in the Released Dataset

• Afrikaans (af)
• Albanian(sq)
• Catalan (ca)
• Czech (cs)
• Danish (da)
• Dutch(nl)
• Estonian (et)
• Filipino (tl)
• Finnish (fi)
• Hebrew (he)
• Hungarian (hu)
• Indonesian (id)
• Italian(it)
• Macedonian (mk)
• Romanian (ro)
• Slovak (sk)
• Swedish (sv)
• Ukrainian (uk)

Bumblebee

The JSONs used by Bumblebee in the XNLI setting can be found under bumblebee_JSONs.

Citation

Please cite the following if you use the data in this repository:

@inproceedings{tan-joty-2021-code-mixing,
    title = "Code-Mixing on Sesame Street: {D}awn of the Adversarial Polyglots",
    author = "Tan, Samson and Joty, Shafiq",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.naacl-main.282",
    pages = "3596--3616",
}