Sentence Similarity

Sentence SImilarity can be used for many domains in natural language processing, including information retrieval, translation memories, question answering etc. There are many methods to calculate the similarity between two strings. In this project I have evaluated those methods using SICK and STS data sets.

Traditional approaches can be found in Traditional Sentence Similarity.ipynb. Methods like edit distance and levenshtein distance have been explored here.

Word Vector approaches can be found in Sentence Similarity - Word Vectors.ipynb. Glove, word2vec and fasttext word embedding models were used for experiements. Distance measures like cosine similarity, word moving distance, smooth inverse frequency were considered.

Context Vector approaches can be found in Sentence Similarity - Elmo Vectors.ipynb and Sentence Similarity - Flair Vectors.ipynb. The same benchmarks in the Word Vector approaches were considered using ELMo, BERT and FLAIR embeddings and compared the results with word2vec embeddings. Results are shown in following tables.

Word Average benchmark

Model	RMSE
AVG-W2V	0.258
AVG-ELMO	0.273
AVG-FLAIR	0.366
AVG-BERT	0.361
AVG-BERT+ELMO	0.334
AVG-W2V-STOP	0.240
AVG-ELMO-STOP	0.263
AVG-FLAIR-STOP	0.299
AVG-BERT-STOP	0.344
AVG-BERT+ELMO-STOP	0.316
AVG-W2V-TFIDF	0.229
AVG-ELMO-TFIDF	0.253
AVG-FLAIR-TFIDF	0.288
AVG-BERT-TFIDF	0.340
AVG-BERT+ELMO-TFIDF	0.309
AVG-W2V-TFIDF-STOP	0.228
AVG-ELMO-TFIDF-STOP	0.249
AVG-FLAIR-TFIDF-STOP	0.269
AVG-BERT-TFIDF-STOP	0.331
AVG-BERT-TFIDF-STOP	0.300

Smooth Inverse Frequency benchmark

Model	RMSE
SIF - W2V ^*	0.204 ^*
SIF - ELMO	0.193
SIF-FLAIR	0.201
SIF-BERT	0.184
SIF-BERT+ELMO^✞	0.181 ^✞

Word Moving distance benchmark

Model	RMSE
WMD-W2V	0.205
WMD-ELMO	0.220
WMD-FLAIR	0.216
WMD-BERT	0.214
WMD-BERT+ELMO	0.218
WMD-W2V-STOP	0.215
WMD-ELM0-STOP	0.238
WMD-FLAIR-STOP	0.224
WMD-BERT-STOP	0.217
WMD-BERT+ELMO-STOP	0.228

Conclusions

Even though the contextual embeddings didn't improve word average and moving distance benchmarks, it improved the smooth inverse frequency benchmark significantly. Best results were provided when BERT and ELMO were stacked together.^✞ denotes the best result and ^* denotes the current best benchmark.

References

If you find this code useful in your research, please consider citing:

@inproceedings{ranasinghe-etal-2019-enhancing,
    title = "Enhancing Unsupervised Sentence Similarity Methods with Deep Contextualised Word Representations",
    author = "Ranasinghe, Tharindu  and
      Orasan, Constantin  and
      Mitkov, Ruslan",
    booktitle = "Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)",
    month = sep,
    year = "2019",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd.",
    url = "https://www.aclweb.org/anthology/R19-1115",
    doi = "10.26615/978-954-452-056-4_115",
    pages = "994--1003",
    abstract = "Calculating Semantic Textual Similarity (STS) plays a significant role in many applications such as question answering, document summarisation, information retrieval and information extraction. All modern state of the art STS methods rely on word embeddings one way or another. The recently introduced contextualised word embeddings have proved more effective than standard word embeddings in many natural language processing tasks. This paper evaluates the impact of several contextualised word embeddings on unsupervised STS methods and compares it with the existing supervised/unsupervised STS methods for different datasets in different languages and different domains",
}
}

aorogat / Simple-Sentence-Similarity