Information Retrieval Engine in Go
Learning project to teach myself Go. Implements an information retrieval engine that supports indexing with four different types of indexes and querying with three different ranking engines.
To build
cd scanner
go build
go install
All functionality is run through a single binary with subcommands. Help for the configuration options is available through a command line flag:
scanner -h
Indexing
The indexer supports building a number of different index types our of the box. These are a standard single-term index, a single-term index with positional posting list, a Porter-stemmed single-term index, and a phrase index which considers ngrams as tokens.
The indexer also supports:
- index pruning to reduce the size of the posting lists created.
See
-index.pruning - in-memory limitations which swap posting lists to disk in an
attempt to reduce active memory usage
See
-memlimit
To run the indexer:
scanner index <args>
Querying
Querying has two components. The query engine, which reads indices and prepares to answer queries, and the querier, which actually makes queries. The query engine should be started first so that it is availbe when the querier is run; it is alos
The connection between the querier and the query engine is implemented with ZeroMQ, because I felt like learning it. Each query can specify the preference for which index should answer the query - if the query is not answered sufficiently by an index, the next preferred query index be used.
To run the query engine:
scanner start-query-engine
To run a query:
scanner query