What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation
Source code, data, and supplementary materials for our EMNLP 2016 article. Please use the following citation:
@inproceedings{Habernal.Gurevych.2016.EMNLP,
title = {What makes a convincing argument? Empirical analysis
and detecting attributes of convincingness in Web argumentation},
author = {Habernal, Ivan and Gurevych, Iryna},
booktitle = {Proceedings of the 2016 Conference on Empirical Methods
in Natural Language Processing},
year = {2016},
pages = {1214--1223},
publisher = {Association for Computational Linguistics},
address = {Austin, Texas},
url = {https://aclweb.org/anthology/D16-1129}
}
Abstract: This article tackles a new challenging task in computational argumentation. Given a pair of two arguments to a certain controversial topic, we aim to directly assess qualitative properties of the arguments in order to explain why one argument is more convincing than the other one. We approach this task in a fully empirical manner by annotating 26k explanations written in natural language. These explanations describe convincingness of arguments in the given argument pair, such as their strengths or flaws. We create a new crowd-sourced corpus containing 9,111 argument pairs, multi-labeled with 17 classes, which was cleaned and curated by employing several strict quality measures. We propose two tasks on this data set, namely (1) predicting the full label distribution and (2) classifying types of flaws in less convincing arguments. Our experiments with feature-rich SVM learners and Bidirectional LSTM neural networks with convolution and attention mechanism reveal that such a novel fine-grained analysis of Web argument convincingness is a very challenging task. We release the new UKPConvArg2 corpus and software under permissive licenses to the research community.
- Contact person: Ivan Habernal, habernal@ukp.informatik.tu-darmstadt.de
- UKP Lab: http://www.ukp.tu-darmstadt.de/
- TU Darmstadt: http://www.tu-darmstadt.de/
Drop me a line or report an issue if something is broken (and shouldn't be) or if you have any questions.
For license information, see LICENSE files in code/*/ and NOTICE.txt.
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.
code— experimental codedata— UKPConvArg2 corpus and data for experiments
The UKPConvArg2 corpus is stored in 32 xml files and is based on the UKPConvArg1 corpus (see our ACL 2016 paper ).
The data are licensed under CC-BY (Creative Commons Attribution 4.0 International License).
The source arguments originate from
- createdebate.com licensed under CC-BY
- convinceme.net licensed under Creative Commons Public Domain License
Here is an excerpt from the is-the-school-uniform-a-good-or-bad-idea-_bad.xml file; most of it is self-explanatory.
<?xml version="1.0"?>
<list>
<annotatedArgumentPair>
<id>arg198954_arg236737</id>
<arg1>
<author>Dilya</author>
<voteUpCount>1</voteUpCount>
<voteDownCount>0</voteDownCount>
<stance>Bad</stance>
<text>I truly believe that wearing uniform is bad, because it ...</text>
<id>arg198954</id>
<originalHTML><p>I truly believe that wearing uniform is bad, because it ...</originalHTML>
</arg1>
<arg2>
<author>debategirl10</author>
<voteUpCount>2</voteUpCount>
<voteDownCount>0</voteDownCount>
<stance>Bad</stance>
<text>I think it is bad to wear school uniform because it ...</text>
<id>arg236737</id>
<originalHTML><p>I think it is bad to wear school uniform because it ...</originalHTML>
</arg2>
<debateMetaData>
<title>Is the school uniform a good or bad idea?</title>
<description/>
<url>http://www.createdebate.com/debate/show/Is_the_school_uniform_a_good_or_bad_idea</url>
</debateMetaData>
<mTurkAssignments>
<mTurkAssignmentWithReasonUnits> <!-- this is taken from the UKPConvArg1 corpus -->
<turkID>A1XJ9VFOJ4LP2X</turkID>
<hitID>3BPP3MA3TCW33YP4JQ191PJF7XUELD</hitID>
<assignmentAcceptTime>2016-02-12 04:08:41.0 UTC</assignmentAcceptTime>
<assignmentSubmitTime>2016-02-12 04:11:44.0 UTC</assignmentSubmitTime>
<value>a1</value>
<reason>a1is well written and intelligent</reason>
<assignmentId>3KRVW3HTZOXR5J90AZRVE3Y3WOTMSA</assignmentId>
<turkRank>1089</turkRank>
<turkCompetence>0.6574247589108307</turkCompetence>
<workerStance>same</workerStance>
<reasonUnits> <!-- these are the new annotations -->
<reasonUnit>
<target>a1</target>
<reasonUnitText>a1is well written and intelligent</reasonUnitText>
<id>41912</id>
<averageCompetenceOfOriginalWorkers>0.6574247589108307</averageCompetenceOfOriginalWorkers>
<!-- this was shown to the crowd-workers -->
<textForAnnotation>Argument Xis well written and intelligent</textForAnnotation>
<assignments> <!-- we have 5 assignments -->
<mTurkReasonUnitAssignment>
<turkID>A29PNROOHHGHDY</turkID>
<hitID>3YCT0L9OMMLU1HX7WY1T3YRKGFZNSI</hitID>
<assignmentId>3ATPCQ38J9MNZ4CKC2TFNYR4RSJYA7</assignmentId>
<assignmentAcceptTime>2016-05-23 19:14:44.0 UTC</assignmentAcceptTime>
<assignmentSubmitTime>2016-05-23 19:16:04.0 UTC</assignmentSubmitTime>
<value>o9_1</value>
<turkCompetence>0.7523819982930656</turkCompetence>
</mTurkReasonUnitAssignment>
<mTurkReasonUnitAssignment>
<turkID>A3EEWSPD9OA2U0</turkID>
<hitID>3YCT0L9OMMLU1HX7WY1T3YRKGFZNSI</hitID>
<assignmentId>3NGMS9VZTMUCV4CXISNJCEVFIX7FF5</assignmentId>
<assignmentAcceptTime>2016-05-23 19:23:25.0 UTC</assignmentAcceptTime>
<assignmentSubmitTime>2016-05-23 19:25:40.0 UTC</assignmentSubmitTime>
<value>o9_1</value>
<turkCompetence>0.5031362986342998</turkCompetence>
</mTurkReasonUnitAssignment>
<mTurkReasonUnitAssignment>
<turkID>AAGKOZ4QSVDZE</turkID>
<hitID>3YCT0L9OMMLU1HX7WY1T3YRKGFZNSI</hitID>
<assignmentId>3II4UPYCOKJCO9OA523N0DW8DL7QD4</assignmentId>
<assignmentAcceptTime>2016-05-23 15:28:42.0 UTC</assignmentAcceptTime>
<assignmentSubmitTime>2016-05-23 15:29:48.0 UTC</assignmentSubmitTime>
<value>o6_2</value>
<turkCompetence>0.004208502893839125</turkCompetence>
</mTurkReasonUnitAssignment>
<mTurkReasonUnitAssignment>
<turkID>AI4J8TH0Y11X5</turkID>
<hitID>3YCT0L9OMMLU1HX7WY1T3YRKGFZNSI</hitID>
<assignmentId>3U5NZHP4LSE80DSXRL7ORTLW0IDHPE</assignmentId>
<assignmentAcceptTime>2016-05-23 21:07:50.0 UTC</assignmentAcceptTime>
<assignmentSubmitTime>2016-05-23 21:08:53.0 UTC</assignmentSubmitTime>
<value>o9_4</value>
<turkCompetence>0.6980055503239371</turkCompetence>
</mTurkReasonUnitAssignment>
<mTurkReasonUnitAssignment>
<turkID>ALNQQ6EXMEII7</turkID>
<hitID>3YCT0L9OMMLU1HX7WY1T3YRKGFZNSI</hitID>
<assignmentId>3B837J3LDP8I2ZMSF7NC02SDABCSRW</assignmentId>
<assignmentAcceptTime>2016-05-23 19:24:15.0 UTC</assignmentAcceptTime>
<assignmentSubmitTime>2016-05-23 19:25:18.0 UTC</assignmentSubmitTime>
<value>o9_1</value>
<turkCompetence>0.5951809237465405</turkCompetence>
</mTurkReasonUnitAssignment>
</assignments>
<estimatedGoldLabel>o9_1</estimatedGoldLabel> <!-- this is the estimated gold label -->
<!-- see below explanation of these labels -->
<!-- some were ignored, some had duplicit text and thus not annotated,
some were filtered out in previous pre-processing phases -->
<ignored>false</ignored>
<duplicate>false</duplicate>
<filtered>false</filtered>
</reasonUnit>
</reasonUnits>
</mTurkAssignmentWithReasonUnits>
....The CSV files are generated from the XML files, here is an excerpt from is-the-school-uniform-a-good-or-bad-idea-_bad.xml.csv
arg198954_arg236737 o8_1,o9_1 I truly believe that ... I think it is bad to wear ...
arg203444_arg251309 o8_1,o9_1,o5_1,o6_3,o7_3 The school my mother works at, plus the school district .. Their gay! Actually this ..
...
- Each line is then a single argument pair, tab-separated
- Pair ID (
firstArgumentID_secondArgumentID) - Comma-delimited set of gold labels as presented in the article in Figure 1
- The more convincing argument
- The less convincing argument
- Line breaks are encoded as
<br/>
- Line breaks are encoded as
- Pair ID (
The labels oA_B correspond to the following description (note that we a bit different notation in the article: CA-B)
o8_1Argument X has more details, information, facts, or examples / more reasons / better reasoning / goes deeper / is more specifico8_4Argument X is balanced, objective, discusses several viewpoints / well-rounded / tackles flaws in opposing viewso8_5Argument X has better credibility / reliability / confidenceo8_6Explanation is highly topic-specific and addresses the content of Argument X in detailo9_1Argument X is clear, crisp, to the point / well writteno9_2Argument X sticks to the topico9_3Argument X has provoking question / makes you thinko9_4Argument X is well thought of / has smart remarks / higher complexityo5_1Argument X is attacking opponent / abusiveo5_2Argument X has language issues / bad grammar / uses humor, jokes, or sarcasmo5_3Argument X is unclear, hard to followo6_1Argument X provides no facts / not enough support / not credible evidence / no clear explanationo6_2Argument X has no reasoning / less or insufficient reasoningo6_3Argument X uses irrelevant reasons / irrelevant informationo7_1Argument X is not an argument / is only opinion / is ranto7_2Argument X is non-sense / has no logical sense / confusingo7_3Argument X is off topic / doesn't address the issueo7_4Argument X is generally weak / vague
Labels o5_*, o6_* and o7_* are always attached to the less convincing argument, while labels o8_* and o9_* to the more convincing argument.
- Java 1.7 and higher, Maven (for Java-based experiments)
- Python 2.7 and virtualenv (for Python-based experiments)
- GPU is recommended but not required
- Tested on 64-bit Linux versions
Installing Python dependencies:
- virtualenv
user@ubuntu:~/emnlp2016-empirical-convincingness$ cd code/src/main/python/
user@ubuntu:~/emnlp2016-empirical-convincingness/code/src/main/python$ virtualenv env
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/user/emnlp2016-empirical-convincingness/code/src/main/python/env/bin/python2
Also creating executable in /home/user/emnlp2016-empirical-convincingness/code/src/main/python/env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.- requirements
user@ubuntu:~/emnlp2016-empirical-convincingness/code/src/main/python$ source env/bin/activate
(env) user@ubuntu:~/emnlp2016-empirical-convincingness/code/src/main/python$ pip install -r requirements.txt
...
Successfully installed Keras-1.0.3 PyYAML-3.11 Theano-0.8.2 nltk-3.1 nose-1.3.7 numpy-1.10.2 scikit-learn-0.17.1 scipy-0.16.1 six-1.10.0-
You should have CUDA installed on your machine for GPU-enabled computation
- Refer to http://deeplearning.net/software/theano/install.html
- This might get sometimes a bit tricky to install
-
Setting-up paths
Adjust path in data-loader.py:
folds, word_index_to_embeddings_map = load_my_data("~/data2/convincingness-emnlp/step14-gold-csv/")so it points to data/CSV-Format
- Run
THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32,optimizer_including=cudnn bidirectional_lstm.py, adjust line 175 to get the right model (MLP, LSTM, ATT+LSTM) - Similarly, run
experiments_single_label.pyfor the second experiment
- Install LIBSVM ( https://www.csie.ntu.edu.tw/~cjlin/libsvm/ version used: Version 3.21, Dec 2015)
- Add
svm-trainandsvm-predictto/usr/local/bin/ - Alternatively, adjust the path constant in SVMLibExperimentRunner
- Add
Compile the Java project
$ cd code/
$ mvn package- Preprocessing: run
de.tudarmstadt.ukp.argumentation.emnlp2016.experiments.preprocessing.Pipeline, then follow similar instructions from https://github.com/UKPLab/acl2016-convincing-arguments