mtuann / nlp-benchmarks

nlp-benchmark

Datasets:

Dataset	Classes	Train samples	Test samples	source
Imdb	2	25 000	25 000	link
AG’s News	4	120 000	7 600	link
Sogou News	5	450 000	60 000	link
DBPedia	14	560 000	70 000	link
Yelp Review Polarity	2	560 000	38 000	link
Yelp Review Full	5	650 000	50 000	link
Yahoo! Answers	10	1 400 000	60 000	link
Amazon Review Full	5	3 000 000	650 000	link
Amazon Review Polarity	2	3 600 000	400 000	link

Models:

• [1]: CNN: Character-level convolutional networks for text classification (paper)
• [2]: VDCNN: Very deep convolutional networks for text classification (paper)
• [3]: HAN: Hierarchical Attention Networks for Document Classification (paper), all credits goes to @cedias
• [4]: Transformer Encoder: Attention Is All You Need (encoder part) (paper), credits to Yu-Hsiang Huang's work)

HAN word (red) and sentence (blue) attention weight at prediction:

Experiments:

Results are reported as follows: (i) / (ii)

• (i): Test set accuracy reported by the paper
• (ii): Test set accuracy reproduced here

Imdb

Model	paper accuracy	repo accuracy
CNN small
VDCNN 9 layers
VDCNN 17 layers
VDCNN 29 layers
HAN		90.5
Transformer		88.6

Ag news

Model	paper accuracy	repo accuracy
CNN small	84.35	88.30
VDCNN 9 layers	90.17	89.22
VDCNN 17 layers	90.61	90.00
VDCNN 29 layers	91.27	90.43
HAN		92.4
Transformer		93.2

Sogu news

Model	paper accuracy	repo accuracy
CNN small	91.35	93.53
VDCNN 9 layers	96.42	93.50
VDCNN 17 layers	96.49
VDCNN 29 layers	96.64	87.90
HAN		96.
Transformer		95.6

DBpedia

Model	paper accuracy	repo accuracy
CNN small	98.02	98.15
VDCNN 9 layers	98.75	98.35
VDCNN 17 layers	98.02	98.15
VDCNN 29 layers	98.71
HAN		99.0
Transformer		98.7

Yelp polarity

Model	paper accuracy	repo accuracy
CNN small
VDCNN 9 layers	94.73	93.97
VDCNN 17 layers	94.95	94.73
VDCNN 29 layers	95.72	94.75
HAN

Yelp review

Model	paper accuracy	repo accuracy
CNN small
VDCNN 9 layers	61.96	61.18
VDCNN 17 layers	62.59
VDCNN 29 layers	64.26	62.73
HAN		63.