This repository contains human (third-party) labels of iSarcasmEval tests sets. We focused on subtasks A and C for both English and Arabic. Each file contains the original label and the aggregated label from the annotators. Each entry was annotated by five annotators. We provide the number of humans who assumed that the text is sarcastic in task A. For task C, we provide the number of votes supporting the majority label.
If you use this data please cite our work:
@inproceedings{abu-farha-2022-sarcasm,
title = "Sarcasm {D}etection is Way Too Easy! {A}n {E}mpirical {C}omparison of {H}uman and {M}achine {S}arcasm Detection",
author = "Abu Farha, Ibrahim and
Wilson, Steven R. and
Oprea, Silviu Vlad and
Magdy, Walid",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
}