Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics
This is the official repository of the NAACL 2022 paper "Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics".
Paper is available at https://arxiv.org/abs/2204.09874.
conda create -n cluster_topic_model python=3.7 -y
conda activate cluster_topic_model
pip install -r requirements.txt
Below is an example of how to run CETopic on the BBC dataset. You can choose a word selecting method from [tfidf_idfi, tfidf_tfi, tfidfi, tfi]
. If you prefer not to reduce the embedding dimensionalities using UMAP, simply set dim_size=-1
. You can train the model, get evaluation results and topics:
from baselines.cetopictm import CETopicTM
from utils import prepare_dataset
dataset, sentences = prepare_dataset('bbc')
tm = CETopicTM(dataset=dataset,
topic_model='cetopic',
num_topics=5,
dim_size=5,
word_select_method='tfidf_idfi',
embedding='princeton-nlp/unsup-simcse-bert-base-uncased',
seed=42)
tm.train()
td_score, cv_score, npmi_score = tm.evaluate()
print(f'td: {td_score} npmi: {npmi_score} cv: {cv_score}')
topics = tm.get_topics()
print(f'Topics: {topics}')
You should expect something similar:
td: 0.96 npmi: 0.11889979828579675 cv: 0.7574707739043192
Topics: {
0: [
('tory', 0.010655754552013494),
('labour', 0.010140645139665033),
('election', 0.008794514704281466),
('party', 0.007523648919704865),
('government', 0.006801391630922592),
('plan', 0.00444844822680986),
('minister', 0.003928431285391421),
('leader', 0.0037359746494665725),
('pension', 0.003697165535497612),
('lord', 0.0036023621214819595)
], ...
}
bash run_evaluate.sh
We use preproessed dataset from OCTIS. You can choose from [bbc, m10, 20ng]
.
You can run our model cetopic
or you can also choose a baseline model from [lda, prodlda, zeroshottm, combinedtm, bertopic]
.
If you use cetopic
, you can also choose a word selecting method from [tfidf_idfi, tfidf_tfi, tfidfi, tfi]
.
You can choose a pretrained model such as princeton-nlp/unsup-simcse-bert-base-uncased
or bert-base-uncased
from SimCSE or HuggingFace.
usage: main.py [-h] [--topic_model TOPIC_MODEL] [--dataset DATASET]
[--pretrained_model PRETRAINED_MODEL] [--num_topics NUM_TOPICS]
[--dim_size DIM_SIZE] [--word_select_method WORD_SELECT_METHOD] [--seed SEED]
Cluster Contextual Embeddings for Topic Models
optional arguments:
-h, --help show this help message and exit
--topic_model TOPIC_MODEL
Topic model to run experiments
--dataset DATASET Datasets to run experiments
--pretrained_model PRETRAINED_MODEL
Pretrained language model
--num_topics NUM_TOPICS
Topic number
--dim_size DIM_SIZE Embedding dimension size to reduce to
--word_select_method WORD_SELECT_METHOD
Word selecting methods to select words from each cluster
--seed SEED Random seed
>> Run cetopic
on BBC dataset using tfidf_idfi
word selecting method and unsupervised SimCSE embeddings, the embedding dimensionality will be reduced to 5 and will ouput 5 topics:
# run_evaluate.sh
DATASET='bbc'
TOPIC_MODEL='cetopic'
WORD_SELECT_METHOD='tfidf_idfi'
PRETRAINED_MODEL='princeton-nlp/unsup-simcse-bert-base-uncased'
python main.py\
--topic_model ${TOPIC_MODEL}\
--dataset ${DATASET}\
--num_topics 5\
--dim_size 5\
--word_select_method ${WORD_SELECT_METHOD}\
--pretrained_model ${PRETRAINED_MODEL}\
--seed 30
>> Run bertopic
on 20NewsGroup dataset using BERT embeddings and expect to ouput 50 topics. Note that BERTopic may not output the exact specified number of topics:
# run_evaluate.sh
DATASET='20ng'
TOPIC_MODEL='bertopic'
PRETRAINED_MODEL='bert-base-uncased'
python main.py\
--topic_model ${TOPIC_MODEL}\
--dataset ${DATASET}\
--num_topics 50\
--pretrained_model ${PRETRAINED_MODEL}\
>> Run combinedtm
on M10 dataset using RoBERTa embeddings and expect to ouput 75 topics:
# run_evaluate.sh
DATASET='m10'
TOPIC_MODEL='combinedtm'
PRETRAINED_MODEL='roberta-base'
python main.py\
--topic_model ${TOPIC_MODEL}\
--dataset ${DATASET}\
--num_topics 75\
--pretrained_model ${PRETRAINED_MODEL}\
To add new topic models, you can inherit the base class TopicModel
and implement your own train()
, evaluate()
, and get_topics()
functions:
class TopicModel:
def __init__(self, dataset, topic_model, num_topics):
self.dataset = dataset
self.topic_model = topic_model
self.num_topics = num_topics
def train(self):
raise NotImplementedError("Train function has not been defined!")
def evaluate(self):
raise NotImplementedError("Evaluate function has not been defined!")
def get_topics(self):
raise NotImplementedError("Get topics function has not been defined!")
If our research helps you, please kindly cite our paper:
@inproceedings{zhang2022neuraltopic,
title={Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics},
author={Zhang, Zihan and Fang, Meng and Chen, Ling and Namazi-Rad, Mohammad-Reza},
booktitle={North American Association for Computational Linguistics (NAACL)},
year={2022}
}
The code is implemented using OCTIS and BERTopic.
MIT