- Search Engine written in Python, with REST API server built using Django framework
- Frontend written in plain HTML, CSS and JS
- Web Crawler written in Python used to gather documents for the Search Engine
The Crawler downloaded over 50k documents from the BBC news website. Documents were preprocessed for the search engine using natural language processing (NLP) methods from the nltk library.
The Search Engine uses sparse matrix technology and bag-of-words representation of downloaded documents, with which it calculates the cosine norm to estimate how much the entered phrase matches a given document. Then it sends to the browser client a predefined number of documents with the greatest correlation score.
SVD matrix decomposition with low rank approximation is used to further reduce the noise in the document dataset and allow the search engine to find documents based on semantic search, not just keyword search.
Libraries:
- numpy
- scipy
- nltk
Frontend was designed to be a Google Search clone with some extra information about the quality of the search engine results.
Note that Django is certainly overkill for a couple of simple REST endpoints. It was used only because I needed to learn it for some other project at that time.