Human Values Detection Behind Arguments
In the current work, we are tackling Task 4 of the Touché Competition: Human Value Detection 2023
Useful resources
| Model | Notebook |
|---|---|
| SVM |  |
| BERT, RoBERTa, DistilBERT | |
| XLNet | |
Task
The task consits of a multilabel text classification. Given a textual argument and a human value category, classify whether or not the argument draws on that category.
Arguments are given as a triplet:
- Conclusion: Conclusion text of the argument
- Stance: Stance of the Premise towards the Conclusion; one of "in favor of", "against"
- Premise: Premise text of the argument
Data
We are using the data available on Zenodo.
We are referring only to the following data: arguments-training.tsv, arguments-validation, labels-training.tsv, labels-validation.tsv
NOTE: Since test data is provided without labels, we did not consider it for our analysis. In this regards the performances of the tested models have been evaluated only on the validation data.
Tested models
- SVM
- BERT-base
- BERT-large
- RoBERTa-base
- RoBERTa-large
- DistilBERT
- XLNet-base
- XLNet-large
Results
Compared to the original paper, the macro-averaged F1 Score has been improved:
- by more 20% for the SVM model
- up to 47% for transformers
| MODEL | SVM | BERT-base | BERT-large | DistilBERT | RoBERTa-base | RoBERTa-large | XLNet-base | XLNet-large |
|---|---|---|---|---|---|---|---|---|
| F1 avg macro (validation) | 0.37 | 0.42 | 0.44 | 0.43 | 0.47 | 0.50 | 0.44 | 0.50 |