This project was carried out as part of the Information Retrieval course.
Implementing a search engine using different search models and algorithms like binary search, tf-idf, and word embeddings. Also, implementing K-means clustering and KNN algorithms to speed up the search.
There are two datasets. One has 7k news articles, and the second one has 50k news articles used for clustering.
Articles in the second dataset have categories (sport, economy, politics, culture, and health), and they were used in KNN for labeling articles in the first dataset. Datasets are not in this repository.
The dataset used in the first and second phase of the project was "IR1_7k_news.xlsx", while "IR00_3_11k News.xlsx" was one of the three datasets used in the third phase. The Hazm library was utilized for Persian language processing.
In the first phase of the project, a boolean information retrieval system was implemented using a positional index. The following preprocessings were carried out:
- Normalization
- Tokenization
- Stemming
The second phase involved the implementation of a ranking information retrieval system based on vector space.
In the third phase, the vector space model was enhanced using word embeddings. Additionally, categorization and clustering were incorporated into the system.