Giters

LuoUndergradXJTU / TwiBot-22

Offical repository of TwiBot-22 @ NeurIPS 2022, Datasets and Benchmarks Track.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

TwiBot-22

This is the official repository of TwiBot-22 @ NeurIPS 2022, Datasets and Benchmarks Track. This dataset is collected from the Twitter website before 2022.

Introduction

TwiBot-22 is the largest and most comprehensive Twitter bot detection benchmark to date. Specifically, TwiBot-22 is designed to address the challenges of limited dataset scale, imcomplete graph structure, and low annotation quality in previous datasets. For more details, please refer to the TwiBot-22 paper and statistics. compare

Dataset Format

Each dataset contains node.json (or tweet.json, user.json, list.json, and hashtag.json for TwiBot-22), label.csv, split.csv and edge.csv (for datasets with graph structure). See here for a detailed description of these files.

How to download TwiBot-22 dataset

TwiBot-22 is available at Google Drive.

Please apply for access by contacting shangbin at cs.washington.edu with your institutional email address and clearly state your institution, your research advisor (if any), and your use case of TwiBot-22.

How to download other datasets

For TwiBot-20, visit the TwiBot-20 github repository.

For other datasets, please visit the Bot Repository.

After downloading these datasets, you can transform them into the 4-file format detailed in "Dataset Format". Alternatively, you can directly download our preprocessed version:

For TwiBot-20, visit the TwiBot-20 github repository, apply for TwiBot-20 access, and there will be a TwiBot-20-Format22.zip in the TwiBot-20 Google Drive link.

For other datasets, you can directly download them from Google Drive. You should adhere to the license of each dataset, the "Content redistribution" section of the Twitter Developer Agreement and Policy, the rules set by the Bot Repository, and only use these datasets for research purposes.

Requirements

  • pip: pip install -r requirements.txt
  • conda : conda install --yes --file requirements.txt

How to run baselines

  1. clone this repo by running git clone https://github.com/LuoUndergradXJTU/TwiBot-22.git
  2. make dataset directory mkdir datasets and download datasets to ./datasets
  3. change directory to src/{name_of_the_baseline}
  4. run experiments under the guidance of corresponding readme.md

Baseline Overview

baseline paper acc on Twibot-22 f1 on Twibot-22 type tags
Abreu et al. link 0.7066 0.5344 F random forest
Alhosseini et al. link 0.4772 0.3810 F G gcn
BGSRD link 0.7188 0.2114 F BERT GAT
Bot Hunter link 0.7279 0.2346 F random forest
Botometer link 0.4987 0.4257 F T G
BotRGCN link 0.7966 0.5750 F T G BotRGCN
Cresci et al. link - - T DNA
Dehghan et al. link - - F T G Graph
Efthimion et al. link 0.7408 0.2758 F T efthimion
EvolveBot link 0.7109 0.1409 F T G random forest
FriendBot link - - F T G random forest
Kipf et al. link 0.7839 0.5496 F T G Graph Neural Network
Velickovic et al. link 0.7948 0.5586 F T G Graph Neural Network
GraphHist link - - F T G random forest
Hayawi et al. link 0.7650 0.2474 F lstm
HGT link 0.7491 0.3960 F T G Graph Neural Networks
SimpleHGN link 0.7672 0.4544 F T G Graph Neural Networks
Kantepe et al. link 0.7640 0.5870 F T random forest
Knauth et al. link 0.7125 0.3709 F T G random forest
Kouvela et al. link 0.7644 0.3003 F T random forest
Kudugunta et al. link 0.6587 0.5167 F SMOTENN, random forest
Lee et al. link 0.7628 0.3041 F T random forest
LOBO link 0.7570 0.3857 F T random forest
Miller et al. link 0.3037 0.4529 F T k means
Moghaddam et al. link 0.7378 0.3207 F G random forest
NameBot link 0.7061 0.0050 F Logistic Regression
RGT link 0.7647 0.4294 F T G Graph Neural Networks
RoBERTa link 0.7207 0.2053 F T RoBERTa
Rodriguez-Ruiz link 0.4936 0.5657 F T G SVM
Santos et al. link - - F T decision tree
SATAR link - - F T G
SGBot link 0.7508 0.3659 F T random forest
T5 link 0.7205 0.2027 T T5
Varol et al. link 0.7392 0.2754 F T random forest
Wei et al. link 0.7020 0.5360 T

where - represents the baseline could not scale to TwiBot-22 dataset

Precision

Precision Botometer-feedback-2019 Cresci-2015 Cresci-2017 Cresci-rtbust-2019 Cresci-stock-2018 Gilani-2017 Midterm-2018 Twibot-20 Twibot-22
Abreu et al. 63.63
$_{3.60}$ 		99.05
$_{0.21}$ 		98.34
$_{0.13}$ 		78.57
$_{1.44}$ 		75.45
$_{0.45}$ 		76.82
$_{1.20}$ 		97.28
$_{0.07}$ 		72.20
$_{0.52}$ 		50.92
$_{0.10}$
Alhosseini et al. - 87.69
$_{1.23}$ 		- - - - - 57.81
$_{0.43}$ 		29.99
$_{3.08}$
BGSRD 27.50
$_{28.2}$ 		86.52
$_{0.64}$ 		75.85
$_{0.00}$ 		58.13
$_{11.1}$ 		52.78
$_{0.75}$ 		25.43
$_{23.2}$ 		84.40
$_{0.93}$ 		67.64
$_{2.26}$ 		22.55
$_{30.9}$
BotHunter - 98.55
$_{0.56}$ 		98.65
$_{0.05}$ 		81.92
$_{2.04}$ 		84.29
$_{0.10}$ 		78.99
$_{0.96}$ 		99.44
$_{0.15}$ 		72.77
$_{0.25}$ 		68.09
$_{0.36}$
Botometer 21.05
-		 50.54
-		 93.35
-		 65.22
-		 68.50
-		 62.99
-		 31.18
-		 55.67
-		 30.81
-
BotRGCN - 95.51
$_{1.02}$ 		- - - - - 84.52
$_{0.54}$ 		74.81
$_{2.22}$
Cresci - 0.59
-		 12.96
-		 - - - - 7.66
-		 -
Dehghan et al. - 96.15
$_{0.00}$ 		- - - - - 94.72
$_{0.00}$ 		-
Efthimion et al. 0.00
$_{0.00}$ 		93.82
$_{0.00}$ 		94.58
$_{0.00}$ 		68.29
$_{0.00}$ 		82.75
$_{0.00}$ 		37.50
$_{0.00}$ 		98.01
$_{0.00}$ 		64.20
$_{0.00}$ 		77.78
$_{0.00}$
EvolveBot - 85.03
$_{3.77}$ 		- - - - - 66.93
$_{0.60}$ 		56.38
$_{0.40}$
FriendBot - 95.29
$_{1.62}$ 		77.55
$_{0.81}$ 		- - - - 72.64
$_{0.52}$ 		-
GCN - 95.59
$_{0.69}$ 		- - - - - 75.23
$_{3.08}$ 		71.19
$_{1.28}$
GAT - 96.10
$_{0.71}$ 		- - - - - 81.39
$_{1.18}$ 		76.23
$_{1.39}$
GraphHist - 73.12
$_{0.10}$ 		- - - - - 51.27
$_{0.20}$ 		-
Hayawi et al. 25.00
$_{0.06}$ 		92.96
$_{0.03}$ 		95.47
$_{0.01}$ 		48.82
$_{0.01}$ 		50.73
$_{0.03}$ 		51.44
$_{0.05}$ 		85.30
$_{0.00}$ 		71.61
$_{0.01}$ 		80.00
$_{0.27}$
HGT - 94.80
$_{0.49}$ 		- - - - - 85.55
$_{0.31}$ 		68.22
$_{2.71}$
SimpleHGN - 95.68
$_{0.90}$ 		- - - - - 84.76
$_{0.46}$ 		72.57
$_{2.79}$
Kantepe et al. - 81.30
$_{1.40}$ 		83.00
$_{0.90}$ 		- - - - 63.40
$_{2.10}$ 		78.60
$_{1.80}$
Knauth et al. 57.41
$_{0.00}$ 		85.70
$_{0.00}$ 		91.56
$_{0.00}$ 		57.41
$_{0.00}$ 		99.89
$_{0.00}$ 		35.17
$_{0.00}$ 		99.91
$_{0.00}$ 		96.56
$_{0.00}$ 		-
Kouvela et al. 48.00
$_{4.47}$ 		99.54
$_{0.18}$ 		99.24
$_{0.13}$ 		82.27
$_{2.00}$ 		82.17
$_{0.46}$ 		79.69
$_{1.09}$ 		97.56
$_{0.04}$ 		79.33
$_{0.44}$ 		69.30
$_{0.14}$
Kudugunta et al. 56.67
$_{10.8}$ 		100.0
$_{0.00}$ 		98.53
$_{0.19}$ 		66.09
$_{2.35}$ 		54.87
$_{0.47}$ 		85.44
$_{2.42}$ 		99.06
$_{0.16}$ 		80.40
$_{0.60}$ 		44.31
$_{0.00}$
Lee et al. 58.97
$_{3.29}$ 		98.65
$_{0.14}$ 		99.56
$_{0.08}$ 		79.37
$_{2.97}$ 		84.75
$_{0.42}$ 		77.58
$_{1.31}$ 		97.36
$_{0.07}$ 		76.60
$_{0.37}$ 		67.23
$_{0.29}$
LOBO - 98.47
$_{0.63}$ 		99.30
$_{0.08}$ 		- - - - 74.83
$_{0.08}$ 		75.43
$_{0.15}$
Miller et al. 0.00
$_{0.00}$ 		72.07
$_{0.00}$ 		77.21
$_{0.18}$ 		52.17
$_{0.00}$ 		54.78
$_{0.00}$ 		48.89
$_{0.00}$ 		83.85
$_{0.00}$ 		60.71
$_{0.20}$ 		29.46
$_{0.00}$
Moghaddam et al. - 98.33
$_{0.26}$ 		- - - - - 72.29
$_{0.67}$ 		67.61
$_{0.10}$
NameBot 45.45
$_{0.00}$ 		76.81
$_{0.00}$ 		80.39
$_{0.03}$ 		65.00
$_{0.00}$ 		58.34
$_{0.00}$ 		58.21
$_{0.00}$ 		86.93
$_{0.00}$ 		58.72
$_{0.00}$ 		67.73
$_{0.00}$
RGT - 96.38
$_{0.59}$ 		- - - - - 85.15
$_{0.28}$ 		75.03
$_{0.85}$
RoBERTa - 97.58
$_{0.27}$ 		92.43
$_{0.99}$ 		- - - - 73.88
$_{1.06}$ 		63.28
$_{0.90}$
Rodriguez-Ruiz et al. - 78.64
$_{0.00}$ 		79.47
$_{0.00}$ 		- - - - 61.60
$_{0.00}$ 		33.23
$_{0.00}$
Santos et al. 50.00
$_{0.00}$ 		72.86
$_{0.00}$ 		81.71
$_{0.00}$ 		75.68
$_{0.00}$ 		65.39
$_{0.00}$ 		32.26
$_{0.00}$ 		88.05
$_{0.00}$ 		62.73
$_{0.00}$ 		-
SATAR - 90.66
$_{0.67}$ 		- - - - - 81.50
$_{1.45}$ 		-
SGBot 59.70
$_{3.91}$ 		99.45
$_{0.20}$ 		98.26
$_{0.17}$ 		83.08
$_{2.60}$ 		83.90
$_{0.29}$ 		82.68
$_{1.88}$ 		99.35
$_{0.22}$ 		76.40
$_{0.40}$ 		73.11
$_{0.18}$
T5 - 91.04
$_{0.29}$ 		94.48
$_{0.65}$ 		- - - - 72.19
$_{0.84}$ 		63.27
$_{0.71}$
Varol et al. - 92.22
$_{0.66}$ 		- - - - - 78.04
$_{0.61}$ 		75.74
$_{0.31}$
Wei et al. - 91.70
$_{1.70}$ 		85.90
$_{1.90}$ 		- - - - 61.00
$_{2.10}$ 		62.70
$_{1.80}$

Recall

Recall Botometer-feedback-2019 Cresci-2015 Cresci-2017 Cresci-rtbust-2019 Cresci-stock-2018 Gilani-2017 Midterm-2018 Twibot-20 Twibot-22
Abreu et al. 46.66
$_{3.00}$ 		62.13
$_{0.97}$ 		91.97
$_{0.69}$ 		89.18
$_{1.40}$ 		75.67
$_{0.73}$ 		58.87
$_{2.75}$ 		98.63
$_{0.08}$ 		82.81
$_{0.51}$ 		11.73
$_{0.06}$
Alhosseini et al. - 97.16
$_{0.81}$ 		- - - - - 95.69
$_{1.93}$ 		56.75
$_{17.69}$
BGSRD 8.57
$_{8.52}$ 		95.56
$_{2.02}$ 		100.0
$_{0.00}$ 		35.14
$_{20.6}$ 		70.40
$_{26.1}$ 		60.00
$_{54.8}$ 		97.66
$_{3.66}$ 		73.19
$_{7.49}$ 		19.90
$_{27.2}$
BotHunter - 91.48
$_{4.16}$ 		85.40
$_{0.19}$ 		83.02
$_{2.95}$ 		79.92
$_{0.54}$ 		62.29
$_{3.47}$ 		99.66
$_{0.06}$ 		86.75
$_{0.46}$ 		14.07
$_{0.12}$
Botometer 57.14
-		 98.95
-		 99.69
-		 100.0
-		 94.96
-		 89.91
-		 87.88
-		 50.82
-		 69.80
-
BotRGCN - 99.17
$_{0.25}$ 		- - - - - 90.19
$_{1.72}$ 		46.80
$_{2.76}$
Cresci - 66.67
-		 95.30
-		 - - - - 67.47
-		 -
Dehghan et al. - 83.88
$_{0.00}$ 		- - - - - 82.19
$_{0.00}$ 		-
Efthimion et al. 0.00
$_{0.00}$ 		94.38
$_{0.00}$ 		89.23
$_{0.00}$ 		75.68
$_{0.00}$ 		58.02
$_{0.00}$ 		2.80
$_{0.00}$ 		94.04
$_{0.00}$ 		70.63
$_{0.00}$ 		16.76
$_{0.00}$
EvolveBot - 95.83
$_{0.66}$ 		- - - - - 72.81
$_{0.41}$ 		8.04
$_{0.05}$
FriendBot - 100.0
$_{0.00}$ 		100.0
$_{0.00}$ 		- - - - 88.94
$_{0.59}$ 		-
GCN - 98.81
$_{0.20}$ 		- - - - - 87.62
$_{3.31}$ 		44.80
$_{1.71}$
GAT - 99.11
$_{0.51}$ 		- - - - - 89.53
$_{0.87}$ 		44.12
$_{1.65}$
GraphHist - 100.0
$_{0.00}$ 		- - - - - 99.05
$_{0.20}$ 		-
Hayawi et al. 17.78
$_{0.06}$ 		79.31
$_{0.02}$ 		92.19
$_{0.03}$ 		81.25
$_{0.09}$ 		71.16
$_{0.07}$ 		28.00
$_{0.13}$ 		98.64
$_{0.00}$ 		83.50
$_{0.04}$ 		14.99
$_{0.05}$
HGT - 99.11
$_{0.12}$ 		- - - - - 91.00
$_{0.57}$ 		28.03
$_{2.60}$
SimpleHGN - 99.29
$_{0.40}$ 		- - - - - 92.06
$_{0.51}$ 		32.90
$_{1.64}$
Kantepe et al. - 75.30
$_{1.20}$ 		76.10
$_{1.10}$ 		- - - - 61.00
$_{1.90}$ 		46.80
$_{1.30}$
Knauth et al. 59.09
$_{0.00}$ 		97.40
$_{0.00}$ 		95.35
$_{0.00}$ 		51.24
$_{0.00}$ 		88.83
$_{0.00}$ 		44.00
$_{0.00}$ 		83.99
$_{0.00}$ 		76.30
$_{0.00}$ 		-
Kouvela et al. 20.00
$_{4.71}$ 		96.79
$_{0.75}$ 		98.98
$_{0.18}$ 		80.00
$_{1.48}$ 		78.78
$_{0.18}$ 		57.20
$_{2.42}$ 		98.92
$_{0.06}$ 		95.17
$_{0.14}$ 		19.17
$_{0.04}$
Kudugunta et al. 45.33
$_{8.69}$ 		60.95
$_{0.21}$ 		85.88
$_{0.37}$ 		50.67
$_{1.21}$ 		47.54
$_{0.60}$ 		35.14
$_{1.70}$ 		90.24
$_{0.66}$ 		33.47
$_{1.30}$ 		61.98
$_{0.00}$
Lee et al. 44.00
$_{3.65}$ 		98.46
$_{0.14}$ 		99.13
$_{0.00}$ 		86.45
$_{1.44}$ 		80.30
$_{0.63}$ 		60.19
$_{2.15}$ 		98.37
$_{0.10}$ 		83.66
$_{0.69}$ 		19.65
$_{0.15}$
LOBO - 99.05
$_{0.13}$ 		96.13
$_{0.39}$ 		- - - - 87.81
$_{0.37}$ 		25.91
$_{0.20}$
Miller et al. 0.00
$_{0.00}$ 		100.0
$_{0.00}$ 		99.11
$_{0.11}$ 		37.50
$_{0.00}$ 		58.89
$_{0.00}$ 		77.19
$_{0.00}$ 		99.81
$_{0.00}$ 		97.44
$_{0.47}$ 		97.89
$_{0.01}$
Moghaddam et al. - 59.23
$_{0.32}$ 		- - - - - 84.38
$_{1.03}$ 		21.02
$_{0.07}$
NameBot 33.33
$_{0.00}$ 		91.12
$_{0.00}$ 		91.79
$_{0.00}$ 		70.27
$_{0.00}$ 		64.13
$_{0.00}$ 		36.45
$_{0.00}$ 		96.82
$_{0.00}$ 		70.47
$_{0.00}$ 		0.03
$_{0.00}$
RGT - 99.23
$_{0.15}$ 		- - - - - 91.06
$_{0.80}$ 		30.10
$_{0.17}$
RoBERTa - 94.11
$_{0.58}$ 		96.27
$_{1.05}$ 		- - - - 72.38
$_{2.05}$ 		12.27
$_{1.22}$
Rodriguez-Ruiz et al. - 99.11
$_{0.00}$ 		92.88
$_{0.00}$ 		- - - - 98.75
$_{0.00}$ 		81.32
$_{0.00}$
Santos et al. 13.33
$_{0.00}$ 		85.80
$_{0.00}$ 		84.40
$_{0.00}$ 		75.68
$_{0.00}$ 		64.95
$_{0.00}$ 		9.35
$_{0.04}$ 		97.24
$_{0.00}$ 		58.13
$_{0.00}$ 		-
SATAR - 99.88
$_{0.16}$ 		- - - - - 91.22
$_{1.82}$ 		-
SGBot 45.33
$_{2.98}$ 		63.67
$_{1.31}$ 		90.86
$_{0.39}$ 		81.62
$_{2.26}$ 		81.03
$_{0.90}$ 		63.62
$_{2.17}$ 		99.66
$_{0.20}$ 		94.91
$_{0.69}$ 		24.32
$_{0.09}$
T5 - 87.71
$_{0.66}$ 		90.26
$_{0.54}$ 		- - - - 69.05
$_{1.46}$ 		12.09
$_{1.43}$
Varol et al. - 97.40
$_{0.90}$ 		- - - - - 84.37
$_{0.67}$ 		16.83
$_{0.21}$
Wei et al. - 75.30
$_{1.50}$ 		72.10
$_{1.50}$ 		- - - - 54.00
$_{2.70}$ 		46.80
$_{1.40}$

F1

F1 Botometer-feedback-2019 Cresci-2015 Cresci-2017 Cresci-rtbust-2019 Cresci-stock-2018 Gilani-2017 Midterm-2018 Twibot-20 Twibot-22
Abreu et al. 53.84
$_{3.03}$ 		76.36
$_{0.72}$ 		95.04
$_{0.30}$ 		83.54
$_{1.04}$ 		76.93
$_{0.58}$ 		66.66
$_{0.10}$ 		97.95
$_{0.03}$ 		77.14
$_{0.46}$ 		53.44
$_{0.09}$
Alhosseini et al. - 92.17
$_{0.36}$ 		- - - - - 72.07
$_{0.48}$ 		38.10
$_{5.93}$
BGSRD 13.03
$_{13.0}$ 		90.80
$_{0.60}$ 		86.27
$_{0.00}$ 		41.08
$_{13.0}$ 		58.18
$_{12.1}$ 		35.72
$_{32.6}$ 		90.50
$_{1.09}$ 		70.05
$_{2.60}$ 		21.14
$_{29.0}$
BotHunter 49.57
$_{3.12}$ 		97.22
$_{0.96}$ 		91.60
$_{3.12}$ 		82.90
$_{1.88}$ 		82.17
$_{0.20}$ 		69.18
$_{1.04}$ 		99.59
$_{0.02}$ 		79.09
$_{0.36}$ 		23.46
$_{0.09}$
Botometer 30.77
-		 66.90
-		 96.12
-		 78.95
-		 79.59
-		 77.39
-		 46.03
-		 53.13
-		 42.75
-
BotRGCN - 97.30
$_{0.53}$ 		- - - - - 87.25
$_{0.73}$ 		57.50
$_{1.42}$
Cresci - 1.17
-		 22.81
-		 - - - - 13.69
-		 -
Dehgan - 88.34
$_{0.00}$ 		- - - - - 76.20
$_{0.00}$ 		-
Efthimion et al. 0.00
$_{0.00}$ 		94.10
$_{0.00}$ 		91.83
$_{0.00}$ 		71.79
$_{0.00}$ 		68.21
$_{0.00}$ 		05.22
$_{0.00}$ 		95.98
$_{0.00}$ 		67.26
$_{0.00}$ 		27.58
$_{0.00}$
EvolveBot - 90.07
$_{1.98}$ 		- - - - - 69.75
$_{0.50}$ 		14.09
$_{0.08}$
FriendBot - 97.58
$_{0.84}$ 		87.35
$_{0.52}$ 		- - - - 79.97
$_{0.34}$ 		-
GCN - 97.17
$_{0.43}$ 		- - - - - 80.86
$_{0.68}$ 		54.96
$_{0.91}$
GAT - 97.58
$_{0.15}$ 		- - - - - 85.25
$_{0.38}$ 		55.86
$_{1.01}$
GraphHist - 84.47
$_{8.23}$ 		- - - - - 67.56
$_{0.30}$ 		-
Hayawi et al. 20.49
$_{0.06}$ 		85.56
$_{0.01}$ 		93.78
$_{0.01}$ 		60.87
$_{0.03}$ 		60.75
$_{0.06}$ 		34.67
$_{0.11}$ 		91.48
$_{0.00}$ 		77.05
$_{0.02}$ 		24.74
$_{0.08}$
HGT - 96.93
$_{0.24}$ 		- - - - - 88.19
$_{0.19}$ 		39.60
$_{2.11}$
SimpleHGN - 97.28
$_{0.39}$ 		- - - - - 88.25
$_{0.18}$ 		45.44
$_{1.65}$
Kantepe et al. - 78.17
$_{1.42}$ 		79.41
$_{1.27}$ 		- - - - 62.23
$_{2.06}$ 		58.71
$_{1.61}$
Knauth et al. 41.27
$_{0.00}$ 		91.18
$_{0.00}$ 		93.42
$_{0.00}$ 		54.15
$_{0.00}$ 		94.03
$_{0.00}$ 		39.10
$_{0.00}$ 		91.26
$_{0.00}$ 		85.24
$_{0.00}$ 		37.09
$_{0.00}$
Kouvela et al. 28.10
$_{5.27}$ 		98.15
$_{0.38}$ 		99.11
$_{0.06}$ 		81.10
$_{1.03}$ 		80.44
$_{0.23}$ 		66.57
$_{1.72}$ 		98.23
$_{0.05}$ 		86.53
$_{0.26}$ 		30.03
$_{0.04}$
Kudugunta et al. 49.61
$_{8.20}$ 		75.74
$_{0.16}$ 		91.74
$_{0.17}$ 		49.22
$_{1.28}$ 		50.94
$_{0.38}$ 		49.75
$_{2.10}$ 		94.45
$_{0.32}$ 		47.26
$_{1.35}$ 		51.67
$_{0.00}$
Lee et al. 50.34
$_{3.16}$ 		98.56
$_{0.11}$ 		99.35
$_{0.04}$ 		82.74
$_{1.79}$ 		82.46
$_{0.36}$ 		67.78
$_{1.81}$ 		97.87
$_{0.07}$ 		79.98
$_{0.50}$ 		30.41
$_{0.20}$
LOBO - 98.76
$_{0.26}$ 		97.69
$_{0.18}$ 		- - - - 80.80
$_{0.20}$ 		38.57
$_{0.23}$
Miller et al. 0.00
$_{0.00}$ 		83.77
$_{0.00}$ 		86.80
$_{0.07}$ 		43.64
$_{0.00}$ 		56.76
$_{0.00}$ 		59.86
$_{0.00}$ 		91.14
$_{0.00}$ 		74.81
$_{0.26}$ 		45.29
$_{0.00}$
Moghaddam et al. - 73.93
$_{0.21}$ 		- - - - - 77.87
$_{0.71}$ 		32.07
$_{0.03}$
NameBot 38.46
$_{0.00}$ 		83.36
$_{0.00}$ 		85.71
$_{0.02}$ 		67.53
$_{0.00}$ 		61.10
$_{0.00}$ 		44.83
$_{0.00}$ 		91.61
$_{0.00}$ 		65.06
$_{0.00}$ 		0.50
$_{0.00}$
RGT - 97.78
$_{0.24}$ 		- - - - - 88.01
$_{0.41}$ 		42.94
$_{1.85}$
RoBERTa - 95.86
$_{0.19}$ 		94.30
$_{0.18}$ 		- - - - 73.09
$_{0.59}$ 		20.53
$_{1.71}$
Rodriguez-Ruiz et al. - 87.70
$_{0.00}$ 		85.65
$_{0.00}$ 		- - - - 63.10
$_{0.00}$ 		56.57
$_{0.00}$
Santos et al. 21.05
$_{0.00}$ 		78.80
$_{0.00}$ 		83.03
$_{0.00}$ 		75.68
$_{0.00}$ 		65.17
$_{0.00}$ 		14.49
$_{0.00}$ 		92.42
$_{0.00}$ 		60.34
$_{0.00}$ 		-
SATAR - 95.05
$_{0.34}$ 		- - - - - 86.07
$_{0.70}$ 		-
SGBot 49.60
$_{3.43}$ 		77.91
$_{0.13}$ 		94.61
$_{0.19}$ 		82.26
$_{1.73}$ 		82.34
$_{0.11}$ 		72.10
$_{0.19}$ 		99.52
$_{0.02}$ 		84.90
$_{0.42}$ 		36.59
$_{0.18}$
T5 - 89.35
$_{0.26}$ 		92.32
$_{0.11}$ 		- - - - 70.57
$_{0.39}$ 		20.27
$_{2.03}$
Varol et al. - 94.73
$_{0.42}$ 		- - - - - 81.08
$_{0.48}$ 		27.54
$_{0.26}$
Wei et al. - 82.65
$_{2.21}$ 		78.43
$_{1.66}$ 		- - - - 57.33
$_{3.19}$ 		53.61
$_{1.36}$

Test1

model Acc F1 precision recall
Moghaddam et al. 89.41
$_{0.30}$ 		24.98
$_{2.72}$ 		16.57
$_{1.97}$ 		50.79
$_{4.25}$
SGBot 91.87
$_{0.11}$ 		47.43
$_{1.21}$ 		76.16
$_{2.31}$ 		34.48
$_{1.56}$
BotHunter 91.44
$_{0.12}$ 		40.39
$_{0.32}$ 		78.28
$_{3.11}$ 		27.24
$_{0.52}$
GAT 91.14
$_{0.45}$ 		47.00
$_{2.92}$ 		64.83
$_{4.31}$ 		36.95
$_{3.04}$
BotRGCN 88.74
$_{0.29}$ 		65.89
$_{1.62}$ 		79.82
$_{2.53}$ 		56.23
$_{3.24}$
RGT 92.8
$_{0.45}$ 		23.39
$_{4.61}$ 		58.33
$_{11.78}$ 		14.66
$_{2.98}$

Test2

model Acc F1 precision recall
Moghaddam et al. 83.93
$_{0.28}$ 		18.49
$_{0.95}$ 		11.58
$_{0.59}$ 		45.94
$_{3.35}$
SGBot 84.72
$_{0.31}$ 		26.00
$_{2.80}$ 		54.55
$_{2.80}$ 		17.11
$_{2.28}$
BotHunter 85.63
$_{0.31}$ 		23.38
$_{1.55}$ 		73.67
$_{9.81}$ 		13.95
$_{1.18}$
GAT 84.93
$_{0.23}$ 		30.47
$_{2.64}$ 		55.64
$_{2.02}$ 		21.05
$_{2.46}$
BotRGCN 85.59
$_{0.68}$ 		55.45
$_{2.77}$ 		67.45
$_{2.74}$ 		47.17
$_{3.65}$
RGT 87.1
$_{1.19}$ 		38.02
$_{7.21}$ 		58.50
$_{10.18}$ 		28.57
$_{6.68}$

Test3

model Acc F1 precision recall
Moghaddam et al. 87.61
$_{0.20}$ 		22.34
$_{1.78}$ 		14.48
$_{1.26}$ 		49.00
$_{2.74}$
SGBot 89.52
$_{0.13}$ 		38.96
$_{1.77}$ 		68.97
$_{1.57}$ 		27.18
$_{1.85}$
BotHunter 89.53
$_{0.12}$ 		33.77
$_{0.45}$ 		76.62
$_{2.45}$ 		21.66
$_{0.25}$
GAT 89.09
$_{0.38}$ 		40.58
$_{2.68}$ 		61.84
$_{3.50}$ 		30.28
$_{2.75}$
BotRGCN 87.92
$_{0.51}$ 		59.46
$_{2.36}$ 		76.88
$_{3.71}$ 		48.66
$_{3.76}$
RGT 89.6
$_{0.72}$ 		26.89
$_{4.71}$ 		56.49
$_{11.34}$ 		18.05
$_{3.63}$

Citation

Please cite TwiBot-22 if you use the TwiBot-22 dataset or this repository

@inproceedings{fengtwibot,
  title={TwiBot-22: Towards Graph-Based Twitter Bot Detection},
  author={Feng, Shangbin and Tan, Zhaoxuan and Wan, Herun and Wang, Ningnan and Chen, Zilong and Zhang, Binchi and Zheng, Qinghua and Zhang, Wenqian and Lei, Zhenyu and Yang, Shujie and others},
  booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track}
}

How to contribute

  1. New dataset: convert the original data to the TwiBot-22 defined schema.
  2. New baseline: load well-formatted dataset from the dataset directory and define your model.

Welcome PR!

Questions?

Feel free to open issues in this repository! Instead of emails, Github issues are much better at facilitating a conversation between you and our team to address your needs. You can also contact Zhaoxuan Tan through tanzhaoxuan at stu.xjtu.edu.cn.

About

Offical repository of TwiBot-22 @ NeurIPS 2022, Datasets and Benchmarks Track.

License:MIT License

Languages

Language:Python 100.0%