This repository demonstrates the architecture of a personalized search system on an ecommerce site, which facilitates delivering personalized user experience in an efficient and scalable way. It contains four parts -
This repository uses amazon review open dataset to perform demo Please follow the link above to setup AWS Glue Crawler and Athena first.
We use AWS Elastic Search as our search engine. It uses bm25 to do keyword matching and ranks results accordingly.
- This notebook demonstrates how to index product information into elastic search engine.
Keyword matching oriented methods could be vulnerable while some content providers perform search engine spamming, i.e. putting irrelevant information in the content in order to getting more exposure than it deserves for its keywords, leading to unsatisfactory search experience. Therefore we need to calculate accurate relevance scores based on semantics and user intentions between contents and search queries. It is known as learning to rank problem.
-
This notebook demonstrates how to use bert to extract the embedding which represents semantic information of contents and search queries and use the embedding as input features to train a ranking model.
-
This notebook uses tf-idf as a baseline of our ranking model, which does not have very good effect.
To optimize user engagement metrics further, personalized ranking is an effective strategy. In this stage, we will users' clickstream data to compile their preferences and calculate the relevance scores between contents and users, and rank the results accordingly.